Nov 02 17:19:58.844344 ip-10-0-136-192 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 17:19:58.844358 ip-10-0-136-192 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 17:19:58.844367 ip-10-0-136-192 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 17:19:58.844690 ip-10-0-136-192 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 17:20:08.990320 ip-10-0-136-192 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 17:20:08.990343 ip-10-0-136-192 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7136298efa6e46f2b81065039c50199f --
Nov 02 17:22:08.647953 ip-10-0-136-192 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:22:09.025646 ip-10-0-136-192 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:09.025646 ip-10-0-136-192 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:22:09.025646 ip-10-0-136-192 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:09.025646 ip-10-0-136-192 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:09.031081 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.030970    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.031081 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031077    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.031081 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031082    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.031081 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031086    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.031081 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031091    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031095    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031099    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031103    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031106    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031110    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031114    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031117    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031121    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031124    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031127    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031131    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031135    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031139    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031142    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031146    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031151    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031155    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031159    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.031400 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031163    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031168    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031172    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031176    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031185    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031189    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031193    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031197    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031201    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031205    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031209    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031213    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031217    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031222    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031226    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031229    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031232    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031236    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031240    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031245    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.032191 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031250    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031254    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031258    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031263    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031267    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031270    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031274    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031278    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031283    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031287    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031291    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031296    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031300    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031304    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031309    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031313    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031318    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031322    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031326    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.033037 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031330    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031336    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031341    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031345    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031349    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031358    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031363    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031368    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031372    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031376    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031380    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031385    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031389    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031393    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031397    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031401    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031405    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031409    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031415    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031419    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.033852 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031423    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031429    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031434    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031438    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031442    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031446    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031453    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031460    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031464    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031468    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031472    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031476    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031483    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031504    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031508    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031512    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031517    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031522    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031528    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.034755 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031534    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031538    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031543    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031547    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.031551    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032910    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032921    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032926    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032930    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032934    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032939    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032943    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032948    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032952    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032956    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032962    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032967    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032971    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032976    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032980    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.035545 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032984    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032988    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032992    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.032996    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033000    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033004    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033008    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033012    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033016    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033021    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033025    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033029    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033033    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033037    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033041    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033045    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033050    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033054    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033058    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033063    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.036429 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033067    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033071    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033076    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033080    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033084    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033088    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033092    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033096    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033102    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033106    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033117    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033125    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033129    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033134    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033138    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033142    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033146    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033150    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.037306 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033154    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033159    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033163    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033167    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033171    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033176    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033180    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033185    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033190    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033194    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033198    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033203    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033207    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033213    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033217    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033221    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033225    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033230    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033234    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033238    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.037918 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033242    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033246    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033252    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033260    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033265    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033269    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033273    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033277    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033281    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033285    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033289    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033293    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033298    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033302    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033307    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033311    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033315    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033321    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033326    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.038653 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033331    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033337    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033343    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033348    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033352    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033355    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033360    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033364    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033369    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033373    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033377    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.033382    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034390    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034403    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034412    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034419    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034424    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034430    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034434    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.039145 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034439    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034443    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034447    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034451    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034456    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034460    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034464    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034469    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034473    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034478    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034482    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034486    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034491    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034495    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034499    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034503    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034507    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034511    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034515    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034519    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.039721 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034523    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034528    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034532    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034536    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034540    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034544    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034549    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034553    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034558    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034562    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034566    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034571    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034575    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034579    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034583    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034587    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034591    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034595    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034599    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034605    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.040200 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034610    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034614    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034618    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034622    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034649    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034653    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034658    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034662    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034666    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034670    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034675    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034679    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034682    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034686    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034690    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034694    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034699    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034703    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034708    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.040706 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034712    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034716    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034722    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034726    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034731    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034735    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034739    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034743    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034748    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034751    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034755    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034760    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034764    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034768    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034772    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034777    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034781    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034786    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034790    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.041150 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034795    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034799    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034803    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034808    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034812    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034816    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034821    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034825    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034829    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034834    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034839    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034843    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034847    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034851    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034859    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034864    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034873    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034877    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034881    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.041605 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.034888    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035039    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035051    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035062    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035070    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035077    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035082    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035090    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035097    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035102    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035107    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035113    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035118    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035123    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035127    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035132    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035137    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035141    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035146    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035153    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035157    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035162    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035167    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035172    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:22:09.042188 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035179    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035184    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035191    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035196    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035201    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035207    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035212    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035217    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035222    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035229    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035234    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035241    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035245    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035251    2577 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035256    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035264    2577 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035269    2577 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035274    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035279    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035284    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035291    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035296    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035301    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035306    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035311    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:22:09.042817 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035315    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035320    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035325    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035329    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035334    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035339    2577 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035345    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035351    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035356    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035363    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035368    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035372    2577 flags.go:64] FLAG: --help="false"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035377    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035393    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035398    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035403    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035408    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035413    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035420    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035425    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035430    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035435    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035440    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:22:09.043499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035445    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035450    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035455    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035459    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035464    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035469    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035473    2577 flags.go:64] FLAG: --lock-file=""
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035478    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035483    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035496    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035513    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035518    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035523    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035529    2577 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035533    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035539    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035543    2577 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035549    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035558    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035563    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035569    2577 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035574    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035579    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035585    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035590    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:22:09.044081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035595    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035600    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035606    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035620    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035642    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035648    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035652    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035657    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035662    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035667    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035671    2577 flags.go:64] FLAG: --port="10250"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035676    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035681    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-019afcce8afbd8359"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035686    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035690    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035695    2577 flags.go:64] FLAG: --register-node="true"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035700    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035707    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035712    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035716    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035721    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035728    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035733    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035738    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035743    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035750    2577 flags.go:64] FLAG: --runonce="false"
Nov 02 17:22:09.044704 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035755    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035760    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035765    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035770    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035774    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035781    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035786    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035791    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035798    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035802    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035807    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035811    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035817    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035822    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035827    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035836    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035841    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035845    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035852    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035857    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035861    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035866    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035871    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035875    2577 flags.go:64] FLAG: --v="2"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035883    2577 flags.go:64] FLAG: --version="false"
Nov 02 17:22:09.045325 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035890    2577 flags.go:64] FLAG: --vmodule=""
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035896    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.035901    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036142    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036151    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036155    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036161    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036166    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036170    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036174    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036178    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036183    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036187    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036192    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036197    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036203    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036208    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036212    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036219    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.045956 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036226    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036231    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036237    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036241    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036246    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036250    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036254    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036258    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036262    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036267    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036271    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036276    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036280    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036285    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036289    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036293    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036297    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036302    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036306    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036312    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.046462 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036317    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036321    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036325    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036329    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036334    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036338    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036342    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036348    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036354    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036358    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036363    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036367    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036371    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036375    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036379    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036383    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036387    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036392    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036396    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036401    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.046962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036405    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036409    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036413    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036418    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036422    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036426    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036430    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036436    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036442    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036446    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036450    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036454    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036458    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036462    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036467    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036471    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036475    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036479    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036484    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.047444 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036488    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036494    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036499    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036503    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036507    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036511    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036515    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036520    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036524    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036528    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036532    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036537    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036541    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036545    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036549    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036553    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036558    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036562    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036566    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.047975 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036570    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036575    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036579    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036583    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036587    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036591    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036596    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036600    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036604    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036608    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036613    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036617    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.036684    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036746    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036753    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036762    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.048424 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036767    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036771    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036775    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036779    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036784    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036788    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036792    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036797    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036801    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036805    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036809    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036813    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036817    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036821    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036825    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036830    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036834    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036839    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036843    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036847    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.048828 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036851    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036856    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036859    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036864    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036869    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036873    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036877    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036881    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036886    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036892    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036896    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036903    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036908    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036913    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036918    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036924    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036928    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036933    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036938    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036942    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.049303 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036946    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036950    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036955    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036959    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036964    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036968    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036972    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036976    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036980    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036985    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036989    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036993    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.036997    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037002    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037006    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037010    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037015    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037019    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037023    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.049843 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037027    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037031    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037035    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037040    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037047    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037051    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037056    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037061    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037065    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037069    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037075    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037079    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037083    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037087    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037092    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037096    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037100    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037104    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037108    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.050283 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037112    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037116    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037121    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037125    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037129    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037133    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037137    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037141    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037145    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037149    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037153    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037157    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037161    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037165    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037172    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037178    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037183    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037189    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037193    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.050745 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037197    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037202    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037207    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037212    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037219    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.037223    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.045759    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.045778    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.045794    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.045799    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045859    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045864    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045868    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045870    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045873    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045876    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045879    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045881    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045884    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045887    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.051193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045889    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045892    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045895    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045897    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045900    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045902    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045905    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045907    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045910    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045913    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045915    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045918    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045920    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045925    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045927    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045930    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045933    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045935    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045939    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045943    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.051720 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045948    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045952    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045954    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045957    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045960    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045963    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045966    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045969    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045972    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045975    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045977    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045980    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045983    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045986    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045988    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045991    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045994    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045996    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.045999    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.052183 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046001    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046004    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046006    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046009    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046011    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046014    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046017    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046020    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046023    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046026    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046028    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046032    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046035    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046038    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046041    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046044    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046046    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046049    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046051    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.052648 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046054    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046056    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046059    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046062    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046064    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046067    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046069    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046072    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046074    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046077    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046080    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046082    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046085    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046088    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046090    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046093    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046095    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046098    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046101    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.053096 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046103    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046106    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046109    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046113    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046116    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046120    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046123    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046125    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046128    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046131    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046133    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046136    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046138    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046141    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046145    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046149    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046152    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046155    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.053582 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046157    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.046190    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046337    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046342    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046345    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046347    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046350    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046353    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046356    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046358    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046361    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046363    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046366    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046370    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046375    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046378    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.054022 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046381    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046385    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046388    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046391    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046394    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046397    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046400    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046403    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046405    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046408    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046410    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046413    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046416    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046418    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046421    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046423    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046426    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046428    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046431    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.054457 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046433    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046436    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046438    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046441    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046443    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046446    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046448    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046451    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046454    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046456    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046459    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046461    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046464    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046466    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046469    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046473    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046476    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046479    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046481    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.055193 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046484    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046487    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046490    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046492    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046495    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046497    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046500    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046502    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046506    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046526    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046530    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046533    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046535    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046538    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046541    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046544    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046546    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046549    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046552    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.055676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046554    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046557    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046559    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046562    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046564    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046567    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046569    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046572    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046575    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046578    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046581    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046584    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046586    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046589    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046591    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046594    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046596    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046599    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046602    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.056199 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046604    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046607    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046609    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046612    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046614    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046617    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046619    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046622    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046643    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046646    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046649    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046652    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046655    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046657    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046660    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046663    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.046686    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:09.056802 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046721    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046725    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046728    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046731    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046734    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046737    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046741    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046746    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046748    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046751    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046754    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046756    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046759    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046761    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046764    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046766    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046769    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046771    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046774    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:09.057202 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046776    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046779    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046781    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046784    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046786    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046789    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046791    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046794    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046797    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046799    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046802    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046804    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046806    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046809    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046811    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046814    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046816    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046819    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046822    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:09.057676 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046824    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046827    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046830    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046832    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046835    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046838    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046840    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046843    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046845    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046847    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046850    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046852    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046855    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046860    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046862    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046865    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046869    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046872    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:09.058114 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046875    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046878    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046880    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046883    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046885    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046888    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046890    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046893    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046896    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046901    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046904    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046906    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046909    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046911    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046913    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046917    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046920    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046922    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046925    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:09.058580 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046927    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046930    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046932    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046934    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046937    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046939    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046942    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046944    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046947    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046950    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046952    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046955    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046957    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046960    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046963    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046965    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046968    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046970    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046974    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046976    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:09.059063 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046979    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046985    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046988    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046992    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046995    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.046997    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.047000    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.047002    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:09.047004    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.047066    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.049048    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.049951    2577 server.go:1014] "Starting client certificate rotation"
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.050054    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.050092    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.051993    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:09.059541 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.056483    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:09.066956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.066938    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:22:09.070128 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.070113    2577 log.go:25] "Validated CRI v1 image API"
Nov 02 17:22:09.070943 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.070914    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:22:09.073508 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.073485    2577 fs.go:135] Filesystem UUIDs: map[076be6ac-4f06-4e2f-a697-40e47d3cb75f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8c8e9bc4-b02e-4ed7-af65-a9cf9fd25127:/dev/nvme0n1p3]
Nov 02 17:22:09.073573 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.073506    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 17:22:09.078564 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.078444    2577 manager.go:217] Machine: {Timestamp:2026-11-02 17:22:09.077328591 +0000 UTC m=+0.314103246 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099301 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f9a2224369cb039ddfc06ed3c908 SystemUUID:ec24f9a2-2243-69cb-039d-dfc06ed3c908 BootID:7136298e-fa6e-46f2-b810-65039c50199f Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:8c:cd:a6:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:8c:cd:a6:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:80:58:c5:e3:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:22:09.078666 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.078565    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:22:09.078697 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.078674    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:22:09.081786 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.081755    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:22:09.081980 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.081788    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-192.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:22:09.082062 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.081985    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:22:09.082062 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.081999    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:22:09.082062 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.082038    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:09.082062 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.082057    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:09.082827 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.082814    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:22:09.086539 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.086523    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:22:09.086711 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.086699    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:22:09.087412 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.087401    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:22:09.087462 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.087420    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:22:09.087462 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.087437    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:22:09.087462 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.087451    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:22:09.087586 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.087462    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:22:09.087855 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.087841    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:09.089904 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.089864    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:22:09.094248 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.094235    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:09.095115 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095102    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:22:09.095146 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095133    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:22:09.095312 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095300    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:22:09.095346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095323    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:22:09.095346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095332    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:22:09.095346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095341    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:22:09.095423 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095351    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:22:09.095423 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095359    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:22:09.095890 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095880    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:22:09.095890 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095891    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:22:09.095956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095898    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:22:09.095956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095905    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:22:09.095956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095914    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:22:09.095956 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.095923    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:22:09.097148 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.097138    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:22:09.097148 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.097150    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:22:09.101021 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.101006    2577 server.go:1263] "Started kubelet"
Nov 02 17:22:09.101137 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.101067    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:22:09.101213 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.101175    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:22:09.101243 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.101233    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:22:09.102392 ip-10-0-136-192 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:22:09.102802 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.102543    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:22:09.105014 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.104980    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:22:09.107143 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.107122    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 17:22:09.109673 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.109654    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:22:09.110171 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.110149    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:22:09.110383 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.110358    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-192.us-west-1.compute.internal\" not found"
Nov 02 17:22:09.110818 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.110800    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:22:09.110896 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.110803    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:22:09.110896 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.110841    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:22:09.111003 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.110913    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:22:09.111003 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.110921    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:22:09.111068 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111061    2577 factory.go:55] Registering systemd factory
Nov 02 17:22:09.111096 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111073    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:22:09.111288 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111275    2577 factory.go:153] Registering CRI-O factory
Nov 02 17:22:09.111324 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111290    2577 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:22:09.111358 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111345    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:22:09.111390 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111365    2577 factory.go:103] Registering Raw factory
Nov 02 17:22:09.111390 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.111380    2577 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:22:09.112208 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.112195    2577 manager.go:319] Starting recovery of all containers
Nov 02 17:22:09.123512 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.123474    2577 manager.go:324] Recovery completed
Nov 02 17:22:09.125164 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.125143    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 02 17:22:09.128617 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.128602    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.130191 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130173    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.130259 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130206    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.130259 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130220    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.130754 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130738    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:22:09.130819 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130753    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:22:09.130819 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130775    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:22:09.130873 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.130746    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.131567 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.131547    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.131660 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.131584    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.131660 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.131597    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.133081 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.133063    2577 policy_none.go:50] "Start"
Nov 02 17:22:09.133149 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.133087    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:22:09.133149 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.133100    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:22:09.135290 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.135279    2577 policy_none.go:44] "Start"
Nov 02 17:22:09.180802 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.180766    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.199610    2577 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.200885    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.200932    2577 server.go:87] "Starting device plugin registration server"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.201283    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.201297    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.201755    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.201838    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.201845    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.203008    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:22:09.223801 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.203071    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-192.us-west-1.compute.internal\" not found"
Nov 02 17:22:09.243942 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.243768    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:22:09.245309 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.245284    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:22:09.245396 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.245318    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:22:09.245396 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.245345    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:22:09.245396 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.245389    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 17:22:09.248062 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.248039    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-192.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 17:22:09.249034 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.248038    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-192.us-west-1.compute.internal.18e44a31bdf30a3a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-192.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-192.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 17:22:09.10097465 +0000 UTC m=+0.337749306,LastTimestamp:2026-11-02 17:22:09.10097465 +0000 UTC m=+0.337749306,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-192.us-west-1.compute.internal,}"
Nov 02 17:22:09.296995 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.296900    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-192.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 17:22:09.302160 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.302144    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.303087 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.303071    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.303159 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.303101    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.303159 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.303113    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.303159 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.303155    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.321374 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.321344    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:22:09.321463 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.321425    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qw5zq"
Nov 02 17:22:09.346889 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.346824    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"]
Nov 02 17:22:09.346984 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.346906    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.347922 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.347906    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.348004 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.347935    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.348004 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.347946    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.350166 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350154    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.350332 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.350365 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350353    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.350969 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350954    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.350969 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350964    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.351070 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350989    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.351070 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.351003    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.351070 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.350990    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.351070 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.351067    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.353186 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.353168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.353261 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.353197    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:09.354046 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.354027    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:09.354127 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.354053    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:09.354127 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.354064    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:09.383597 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.383577    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-192.us-west-1.compute.internal\" not found" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.385599 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.385582    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-192.us-west-1.compute.internal\" not found" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.402954 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.402925    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:09.404329 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.404307    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:09.412432 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.412406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.412537 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.412435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.412537 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.412455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2661666f349d448cd741b0565fc226b6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"2661666f349d448cd741b0565fc226b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.501642 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.501612    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:09.503576 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.503556    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:09.504035 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.504012    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.504143 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:09.504039    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-192.us-west-1.compute.internal\": node \"ip-10-0-136-192.us-west-1.compute.internal\" not found"
Nov 02 17:22:09.510640 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.510605    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.512818 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.512800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.512905 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.512824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2661666f349d448cd741b0565fc226b6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"2661666f349d448cd741b0565fc226b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.512905 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.512843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.512905 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.512899    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.512998 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.512905    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2661666f349d448cd741b0565fc226b6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"2661666f349d448cd741b0565fc226b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.512998 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.512899    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.567122 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.567059    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:09.567724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.567710    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.625884 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.625854    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:09.686665 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.686621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:09.688936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:09.688905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:10.050454 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.050359    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 17:22:10.050972 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.050587    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:22:10.050972 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.050597    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:22:10.050972 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.050644    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:22:10.050972 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.050676    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://acf688e646e8844ffb176668e16206eb-5f8c15bb6010e1e3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-192.us-west-1.compute.internal.18e44a31bfb163ad\": read tcp 10.0.136.192:52316->18.220.170.94:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-192.us-west-1.compute.internal.18e44a31bfb163ad  default   4686 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-192.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-192.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-192.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 17:22:09 +0000 UTC,LastTimestamp:2026-11-02 17:22:09.303116672 +0000 UTC m=+0.539891328,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-192.us-west-1.compute.internal,}"
Nov 02 17:22:10.088441 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.088413    2577 apiserver.go:52] "Watching apiserver"
Nov 02 17:22:10.110515 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.110481    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 17:22:10.326185 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.326082    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:17:10 +0000 UTC" deadline="2028-04-09 10:37:17.317999136 +0000 UTC"
Nov 02 17:22:10.326185 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.326127    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12569h15m6.991875813s"
Nov 02 17:22:10.355289 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.355256    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:22:10.356241 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.356190    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-pmr9j","kube-system/global-pull-secret-syncer-trcmz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal","openshift-dns/node-resolver-hs745","openshift-image-registry/node-ca-nqt8d","openshift-multus/multus-additional-cni-plugins-9spt5","openshift-multus/network-metrics-daemon-2b8zt","openshift-ovn-kubernetes/ovnkube-node-mhl5c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj","openshift-multus/multus-4df8n","openshift-network-diagnostics/network-check-target-slw2l","openshift-network-operator/iptables-alerter-tw9mq","kube-system/konnectivity-agent-dz5j7"]
Nov 02 17:22:10.359350 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.359326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.361439 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.361415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.361546 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.361512    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:10.363497 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.363469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.365551 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.365533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.367766 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.367744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.369884 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.369865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:10.369978 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.369932    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:10.372115 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.372092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.374324 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.374302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.376574 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.376552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.379130 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.379115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:10.379216 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.379199    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:10.381270 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.381254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.383785 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.383766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.410998 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.410975    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.411154 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.411134    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.411315 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.411301    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-546w9\""
Nov 02 17:22:10.411765 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.411753    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:22:10.414806 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.414789    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.414871 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.414854    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.415023 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.415008    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkkxw\""
Nov 02 17:22:10.416770 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.416754    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kdpj\""
Nov 02 17:22:10.416857 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.416844    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:22:10.416936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.416921    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.416971 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.416934    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.417026 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-node-log\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417059 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-device-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.417059 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysconfig\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.417135 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-sys\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.417135 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-var-lib-kubelet\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.417233 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9867d6a5-3527-4058-80a0-1d8dd1e1590a-serviceca\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.417233 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-slash\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417338 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-system-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417338 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-bin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417338 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-var-lib-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417479 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cni-binary-copy\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417479 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-run\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.417479 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qtn\" (UniqueName: \"kubernetes.io/projected/9867d6a5-3527-4058-80a0-1d8dd1e1590a-kube-api-access-j9qtn\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.417479 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-ovn\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cnibin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-conf-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6qp\" (UniqueName: \"kubernetes.io/projected/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-kube-api-access-7l6qp\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-tmp\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.417685 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ac25cf-7824-4fca-8c91-d3c7293e3834-tmp-dir\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-systemd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-env-overrides\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29109340-d347-45c7-a6dc-891cc6f752a9-ovn-node-metrics-cert\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.417936 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-netns\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-dbus\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9ac25cf-7824-4fca-8c91-d3c7293e3834-hosts-file\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.417989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-netd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-system-cni-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-socket-dir-parent\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-kubelet\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-kubernetes\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-systemd\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cnibin\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.418256 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-registration-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jfb\" (UniqueName: \"kubernetes.io/projected/efe751b0-e2fe-4576-83de-e9dcb258291b-kube-api-access-w6jfb\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxx2t\" (UniqueName: \"kubernetes.io/projected/6a5b0c75-fe80-4472-b423-97785579f253-kube-api-access-lxx2t\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-systemd-units\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-log-socket\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-config\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-socket-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-multus-certs\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efe751b0-e2fe-4576-83de-e9dcb258291b-iptables-alerter-script\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppnh\" (UniqueName: \"kubernetes.io/projected/00939ab6-259b-42cd-be51-95182c9ba479-kube-api-access-hppnh\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-etc-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-daemon-config\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.418727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-etc-kubernetes\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-conf\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jd7\" (UniqueName: \"kubernetes.io/projected/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kube-api-access-c6jd7\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-multus\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efe751b0-e2fe-4576-83de-e9dcb258291b-host-slash\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9vq\" (UniqueName: \"kubernetes.io/projected/c9ac25cf-7824-4fca-8c91-d3c7293e3834-kube-api-access-lc9vq\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-kubelet\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-netns\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.418982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-bin\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-kubelet-config\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-hostroot\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.419346 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f80bf07-9f23-46af-b53f-35bbd3157b6d-agent-certs\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f80bf07-9f23-46af-b53f-35bbd3157b6d-konnectivity-ca\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-lib-modules\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-host\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-etc-tuned\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9867d6a5-3527-4058-80a0-1d8dd1e1590a-host\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f548\" (UniqueName: \"kubernetes.io/projected/29109340-d347-45c7-a6dc-891cc6f752a9-kube-api-access-9f548\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsqh\" (UniqueName: \"kubernetes.io/projected/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-kube-api-access-mcsqh\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-modprobe-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-script-lib\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-sys-fs\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.419895 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.419467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-k8s-cni-cncf-io\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.420522 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.420504    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:22:10.420596 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.420530    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:22:10.420596 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.420537    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.420596 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.420559    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.420727 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.420646    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2rxq\""
Nov 02 17:22:10.427697 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.427678    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.427697 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.427677    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:22:10.428131 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.428116    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:22:10.428182 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.428166    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:22:10.456490 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.456462    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.456490 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.456492    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jxqfl\""
Nov 02 17:22:10.456700 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.456501    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:22:10.456700 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.456492    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.460752 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.460720    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:22:10.460752 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.460738    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:22:10.460959 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.460765    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztdmt\""
Nov 02 17:22:10.460959 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.460809    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4464\""
Nov 02 17:22:10.461060 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.461046    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.461111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.461094    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:22:10.464910 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.464889    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:10.464910 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.464889    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:10.465060 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.464966    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:22:10.470302 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.470266    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:22:10.470397 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.470350    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wf5x\""
Nov 02 17:22:10.470454 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.470415    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-867zk\""
Nov 02 17:22:10.470454 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.470428    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:22:10.477920 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.477895    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:22:10.478035 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.477969    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j57sg"
Nov 02 17:22:10.519778 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-netd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-system-cni-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-socket-dir-parent\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-system-cni-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-netd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519865    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-socket-dir-parent\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-kubelet\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-kubernetes\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519918    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-kubelet\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.519933 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-systemd\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cnibin\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519955    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-kubernetes\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519984    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-systemd\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.519982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-registration-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cnibin\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jfb\" (UniqueName: \"kubernetes.io/projected/efe751b0-e2fe-4576-83de-e9dcb258291b-kube-api-access-w6jfb\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-registration-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxx2t\" (UniqueName: \"kubernetes.io/projected/6a5b0c75-fe80-4472-b423-97785579f253-kube-api-access-lxx2t\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-systemd-units\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-log-socket\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-config\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-socket-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-multus-certs\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efe751b0-e2fe-4576-83de-e9dcb258291b-iptables-alerter-script\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppnh\" (UniqueName: \"kubernetes.io/projected/00939ab6-259b-42cd-be51-95182c9ba479-kube-api-access-hppnh\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:10.520245 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-etc-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-daemon-config\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-etc-kubernetes\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-conf\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jd7\" (UniqueName: \"kubernetes.io/projected/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kube-api-access-c6jd7\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520333    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-socket-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-multus\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520370    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520378    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-systemd-units\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efe751b0-e2fe-4576-83de-e9dcb258291b-host-slash\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520417    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efe751b0-e2fe-4576-83de-e9dcb258291b-host-slash\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520420    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-etc-kubernetes\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9vq\" (UniqueName: \"kubernetes.io/projected/c9ac25cf-7824-4fca-8c91-d3c7293e3834-kube-api-access-lc9vq\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-kubelet\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-netns\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-bin\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-kubelet-config\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-conf\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-hostroot\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520590    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-log-socket\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f80bf07-9f23-46af-b53f-35bbd3157b6d-agent-certs\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-multus\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f80bf07-9f23-46af-b53f-35bbd3157b6d-konnectivity-ca\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-lib-modules\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-host\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-etc-tuned\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.521965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9867d6a5-3527-4058-80a0-1d8dd1e1590a-host\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f548\" (UniqueName: \"kubernetes.io/projected/29109340-d347-45c7-a6dc-891cc6f752a9-kube-api-access-9f548\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsqh\" (UniqueName: \"kubernetes.io/projected/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-kube-api-access-mcsqh\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-modprobe-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-script-lib\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-sys-fs\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-k8s-cni-cncf-io\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-daemon-config\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.520977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-node-log\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-device-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521041    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysconfig\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521188    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efe751b0-e2fe-4576-83de-e9dcb258291b-iptables-alerter-script\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-lib-modules\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521280    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-multus-certs\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f80bf07-9f23-46af-b53f-35bbd3157b6d-konnectivity-ca\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9867d6a5-3527-4058-80a0-1d8dd1e1590a-host\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-bin\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.522724 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-etc-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521763    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-k8s-cni-cncf-io\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521764    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-script-lib\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-modprobe-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-sys-fs\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521903    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-node-log\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.521999    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-kubelet-config\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-hostroot\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.522056    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret podName:6f939783-c4a2-4931-9ab9-07f56e14a7ed nodeName:}" failed. No retries permitted until 2026-11-02 17:22:11.022038724 +0000 UTC m=+2.258813386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret") pod "global-pull-secret-syncer-trcmz" (UID: "6f939783-c4a2-4931-9ab9-07f56e14a7ed") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522060    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-kubelet\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522096    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-device-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.521046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysconfig\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-netns\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.523533 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-sys\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-var-lib-kubelet\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9867d6a5-3527-4058-80a0-1d8dd1e1590a-serviceca\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-slash\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-host\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-system-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-bin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-var-lib-kubelet\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-var-lib-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cni-binary-copy\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-run\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qtn\" (UniqueName: \"kubernetes.io/projected/9867d6a5-3527-4058-80a0-1d8dd1e1590a-kube-api-access-j9qtn\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522346    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-sys\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522349    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-system-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-ovn\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-slash\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-var-lib-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.524417 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cnibin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522447    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-run\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-conf-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-bin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6qp\" (UniqueName: \"kubernetes.io/projected/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-kube-api-access-7l6qp\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cnibin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-tmp\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ac25cf-7824-4fca-8c91-d3c7293e3834-tmp-dir\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-ovn\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-systemd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522823    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9867d6a5-3527-4058-80a0-1d8dd1e1590a-serviceca\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522978    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cni-binary-copy\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-conf-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525282 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.522829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ac25cf-7824-4fca-8c91-d3c7293e3834-tmp-dir\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-env-overrides\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29109340-d347-45c7-a6dc-891cc6f752a9-ovn-node-metrics-cert\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-netns\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523244    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-systemd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-dbus\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9ac25cf-7824-4fca-8c91-d3c7293e3834-hosts-file\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9ac25cf-7824-4fca-8c91-d3c7293e3834-hosts-file\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-netns\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.523420    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.523498    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:11.023471128 +0000 UTC m=+2.260245777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.523549    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-dbus\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:10.525800 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.524367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-env-overrides\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.526252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.524432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.526252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.524551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-config\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.526252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.525284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-etc-tuned\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.526252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.525412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-tmp\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.526252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.525466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f80bf07-9f23-46af-b53f-35bbd3157b6d-agent-certs\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.526252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.525613    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29109340-d347-45c7-a6dc-891cc6f752a9-ovn-node-metrics-cert\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.555026 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.555005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:10.577413 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.577333    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:10.577413 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.577368    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:10.577413 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.577383    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r455q for pod openshift-network-diagnostics/network-check-target-slw2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:10.577726 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:10.577473    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q podName:25000ba3-538e-482f-bec4-e882e1053b1f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:11.077449794 +0000 UTC m=+2.314224456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r455q" (UniqueName: "kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q") pod "network-check-target-slw2l" (UID: "25000ba3-538e-482f-bec4-e882e1053b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:10.578465 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.578432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jd7\" (UniqueName: \"kubernetes.io/projected/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kube-api-access-c6jd7\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.578567 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.578518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxx2t\" (UniqueName: \"kubernetes.io/projected/6a5b0c75-fe80-4472-b423-97785579f253-kube-api-access-lxx2t\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.578653 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.578617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f548\" (UniqueName: \"kubernetes.io/projected/29109340-d347-45c7-a6dc-891cc6f752a9-kube-api-access-9f548\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.578983 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.578960    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsqh\" (UniqueName: \"kubernetes.io/projected/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-kube-api-access-mcsqh\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.579085 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.579042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jfb\" (UniqueName: \"kubernetes.io/projected/efe751b0-e2fe-4576-83de-e9dcb258291b-kube-api-access-w6jfb\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.579823 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.579803    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6qp\" (UniqueName: \"kubernetes.io/projected/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-kube-api-access-7l6qp\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.579913 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.579882    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9vq\" (UniqueName: \"kubernetes.io/projected/c9ac25cf-7824-4fca-8c91-d3c7293e3834-kube-api-access-lc9vq\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.580816 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.580796    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppnh\" (UniqueName: \"kubernetes.io/projected/00939ab6-259b-42cd-be51-95182c9ba479-kube-api-access-hppnh\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:10.581000 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.580984    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qtn\" (UniqueName: \"kubernetes.io/projected/9867d6a5-3527-4058-80a0-1d8dd1e1590a-kube-api-access-j9qtn\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.708788 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.708745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:10.714566 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.714543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:10.745017 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.744984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:10.773525 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.773499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:10.774456 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.774425    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9867d6a5_3527_4058_80a0_1d8dd1e1590a.slice/crio-c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77 WatchSource:0}: Error finding container c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77: Status 404 returned error can't find the container with id c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77
Nov 02 17:22:10.774962 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.774925    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab93effd0670cefe2422e74f59f96c7d.slice/crio-c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d WatchSource:0}: Error finding container c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d: Status 404 returned error can't find the container with id c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d
Nov 02 17:22:10.775867 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.775846    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2661666f349d448cd741b0565fc226b6.slice/crio-258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03 WatchSource:0}: Error finding container 258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03: Status 404 returned error can't find the container with id 258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03
Nov 02 17:22:10.779020 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.778941    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:22:10.784574 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.784551    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a79d8cf_3095_4d18_9cc4_9aa8dc6ef57f.slice/crio-295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f WatchSource:0}: Error finding container 295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f: Status 404 returned error can't find the container with id 295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f
Nov 02 17:22:10.793991 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.793970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:10.803473 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.803447    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29109340_d347_45c7_a6dc_891cc6f752a9.slice/crio-235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e WatchSource:0}: Error finding container 235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e: Status 404 returned error can't find the container with id 235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e
Nov 02 17:22:10.804478 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.804460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:10.813003 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.812979    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd70ad9e_6914_4bba_8e93_de02ed571a4f.slice/crio-6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66 WatchSource:0}: Error finding container 6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66: Status 404 returned error can't find the container with id 6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66
Nov 02 17:22:10.820362 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.820341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4df8n"
Nov 02 17:22:10.829920 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.829868    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f98f6e_7d08_4d15_9df2_24d5c205a5d3.slice/crio-35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80 WatchSource:0}: Error finding container 35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80: Status 404 returned error can't find the container with id 35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80
Nov 02 17:22:10.852374 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:10.852339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:10.861925 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:10.861888    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe751b0_e2fe_4576_83de_e9dcb258291b.slice/crio-087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b WatchSource:0}: Error finding container 087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b: Status 404 returned error can't find the container with id 087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b
Nov 02 17:22:11.012812 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.012776    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:11.027651 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.027619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:11.027858 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.027734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:11.027858 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.027798    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:11.027858 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.027836    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:11.027975 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.027872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret podName:6f939783-c4a2-4931-9ab9-07f56e14a7ed nodeName:}" failed. No retries permitted until 2026-11-02 17:22:12.027856434 +0000 UTC m=+3.264631078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret") pod "global-pull-secret-syncer-trcmz" (UID: "6f939783-c4a2-4931-9ab9-07f56e14a7ed") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:11.027975 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.027889    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:12.02788149 +0000 UTC m=+3.264656135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:11.128807 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.128696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:11.129495 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.128897    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:11.129495 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.128921    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:11.129495 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.128936    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r455q for pod openshift-network-diagnostics/network-check-target-slw2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:11.129495 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:11.129003    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q podName:25000ba3-538e-482f-bec4-e882e1053b1f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:12.12898537 +0000 UTC m=+3.365760015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r455q" (UniqueName: "kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q") pod "network-check-target-slw2l" (UID: "25000ba3-538e-482f-bec4-e882e1053b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:11.233680 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.233629    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:11.251192 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.251127    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"ab93effd0670cefe2422e74f59f96c7d","Type":"ContainerStarted","Data":"c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d"}
Nov 02 17:22:11.252839 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.252799    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"2661666f349d448cd741b0565fc226b6","Type":"ContainerStarted","Data":"258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03"}
Nov 02 17:22:11.253972 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.253947    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tw9mq" event={"ID":"efe751b0-e2fe-4576-83de-e9dcb258291b","Type":"ContainerStarted","Data":"087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b"}
Nov 02 17:22:11.254989 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.254968    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4df8n" event={"ID":"55f98f6e-7d08-4d15-9df2-24d5c205a5d3","Type":"ContainerStarted","Data":"35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80"}
Nov 02 17:22:11.255964 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.255947    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj" event={"ID":"cd70ad9e-6914-4bba-8e93-de02ed571a4f","Type":"ContainerStarted","Data":"6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66"}
Nov 02 17:22:11.257130 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.257109    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e"}
Nov 02 17:22:11.258180 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.258157    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerStarted","Data":"295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f"}
Nov 02 17:22:11.259181 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.259161    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqt8d" event={"ID":"9867d6a5-3527-4058-80a0-1d8dd1e1590a","Type":"ContainerStarted","Data":"c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77"}
Nov 02 17:22:11.404239 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.404213    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:11.406054 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:11.406029    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f80bf07_9f23_46af_b53f_35bbd3157b6d.slice/crio-cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3 WatchSource:0}: Error finding container cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3: Status 404 returned error can't find the container with id cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3
Nov 02 17:22:11.480818 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.480759    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:11 +0000 UTC" deadline="2028-06-01 01:02:07.716221674 +0000 UTC"
Nov 02 17:22:11.480818 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:11.480811    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13831h39m56.235413638s"
Nov 02 17:22:11.551054 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:11.551025    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5b0c75_fe80_4472_b423_97785579f253.slice/crio-615868fa7c6fddabc214d2a8783d496f72ce561f9d46a2d2b3a0489dd721afb0 WatchSource:0}: Error finding container 615868fa7c6fddabc214d2a8783d496f72ce561f9d46a2d2b3a0489dd721afb0: Status 404 returned error can't find the container with id 615868fa7c6fddabc214d2a8783d496f72ce561f9d46a2d2b3a0489dd721afb0
Nov 02 17:22:11.676225 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:11.676179    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ac25cf_7824_4fca_8c91_d3c7293e3834.slice/crio-7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9 WatchSource:0}: Error finding container 7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9: Status 404 returned error can't find the container with id 7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9
Nov 02 17:22:12.035141 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.035038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:12.035141 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.035132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:12.035374 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.035307    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:12.035440 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.035376    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:14.035356765 +0000 UTC m=+5.272131430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:12.035510 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.035445    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:12.035510 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.035477    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret podName:6f939783-c4a2-4931-9ab9-07f56e14a7ed nodeName:}" failed. No retries permitted until 2026-11-02 17:22:14.035466775 +0000 UTC m=+5.272241434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret") pod "global-pull-secret-syncer-trcmz" (UID: "6f939783-c4a2-4931-9ab9-07f56e14a7ed") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:12.135508 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.135463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:12.136061 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.135697    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:12.136061 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.135720    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:12.136061 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.135734    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r455q for pod openshift-network-diagnostics/network-check-target-slw2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:12.136061 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.135797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q podName:25000ba3-538e-482f-bec4-e882e1053b1f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:14.135776655 +0000 UTC m=+5.372551316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r455q" (UniqueName: "kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q") pod "network-check-target-slw2l" (UID: "25000ba3-538e-482f-bec4-e882e1053b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:12.245993 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.245955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:12.246204 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.246109    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:12.246204 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.246197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:12.246334 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.246288    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:12.246391 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.246353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:12.246451 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:12.246407    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:12.270004 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.269948    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dz5j7" event={"ID":"3f80bf07-9f23-46af-b53f-35bbd3157b6d","Type":"ContainerStarted","Data":"cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3"}
Nov 02 17:22:12.274608 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.274572    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hs745" event={"ID":"c9ac25cf-7824-4fca-8c91-d3c7293e3834","Type":"ContainerStarted","Data":"7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9"}
Nov 02 17:22:12.278006 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.277971    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmr9j" event={"ID":"6a5b0c75-fe80-4472-b423-97785579f253","Type":"ContainerStarted","Data":"615868fa7c6fddabc214d2a8783d496f72ce561f9d46a2d2b3a0489dd721afb0"}
Nov 02 17:22:12.481234 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.481123    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:11 +0000 UTC" deadline="2028-04-04 02:23:58.179678171 +0000 UTC"
Nov 02 17:22:12.481234 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:12.481184    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12441h1m45.69850014s"
Nov 02 17:22:14.058728 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:14.058625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:14.058728 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:14.058723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:14.059584 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.058901    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:14.059584 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.058972    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret podName:6f939783-c4a2-4931-9ab9-07f56e14a7ed nodeName:}" failed. No retries permitted until 2026-11-02 17:22:18.058953505 +0000 UTC m=+9.295728152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret") pod "global-pull-secret-syncer-trcmz" (UID: "6f939783-c4a2-4931-9ab9-07f56e14a7ed") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:14.059584 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.059395    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:14.059584 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.059453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:18.059435508 +0000 UTC m=+9.296210157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:14.160782 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:14.160312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:14.160782 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.160577    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:14.160782 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.160600    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:14.160782 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.160616    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r455q for pod openshift-network-diagnostics/network-check-target-slw2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:14.160782 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.160680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q podName:25000ba3-538e-482f-bec4-e882e1053b1f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:18.160661023 +0000 UTC m=+9.397435676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r455q" (UniqueName: "kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q") pod "network-check-target-slw2l" (UID: "25000ba3-538e-482f-bec4-e882e1053b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:14.246585 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:14.246545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:14.246798 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.246708    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:14.247364 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:14.247110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:14.247364 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.247225    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:14.247364 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:14.247287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:14.247364 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:14.247356    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:15.305120 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:15.299715    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerStarted","Data":"2c482d5d22369a9dc63a412c8ba11dacd07eddd9945ca3d0efc23551d6e6e462"}
Nov 02 17:22:16.247846 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.246622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:16.247846 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:16.247051    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:16.247846 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.247475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:16.247846 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:16.247602    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:16.247846 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.247662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:16.247846 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:16.247751    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:16.306701 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.305713    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="2c482d5d22369a9dc63a412c8ba11dacd07eddd9945ca3d0efc23551d6e6e462" exitCode=0
Nov 02 17:22:16.306701 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.305821    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerDied","Data":"2c482d5d22369a9dc63a412c8ba11dacd07eddd9945ca3d0efc23551d6e6e462"}
Nov 02 17:22:16.313341 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.313305    2577 generic.go:356] "Generic (PLEG): container finished" podID="ab93effd0670cefe2422e74f59f96c7d" containerID="7312ddb41b161c089c167135c7936b922429a552bce0af2615e978ca1360ecf1" exitCode=0
Nov 02 17:22:16.313466 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:16.313350    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"ab93effd0670cefe2422e74f59f96c7d","Type":"ContainerDied","Data":"7312ddb41b161c089c167135c7936b922429a552bce0af2615e978ca1360ecf1"}
Nov 02 17:22:17.318398 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:17.318353    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dz5j7" event={"ID":"3f80bf07-9f23-46af-b53f-35bbd3157b6d","Type":"ContainerStarted","Data":"da2f18601cfaedcf9f98dd8edaa19dd0084fc367a80c79ec4b8e83b17e75548c"}
Nov 02 17:22:17.329872 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:17.329130    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"ab93effd0670cefe2422e74f59f96c7d","Type":"ContainerStarted","Data":"9ead4309da96b5fa395b6d2500970a2c0cd325c34ce533dbd3560bf0c47eaf59"}
Nov 02 17:22:17.336313 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:17.335830    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"2661666f349d448cd741b0565fc226b6","Type":"ContainerStarted","Data":"e6b0eab0da957b8535801ed79eb9c38bf5d432fffad1ea99b81c680818310926"}
Nov 02 17:22:17.438320 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:17.438209    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dz5j7" podStartSLOduration=2.446503864 podStartE2EDuration="7.438188357s" totalImagesPullingTime="4.991684493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:17.437912956 +0000 UTC m=+8.674687626" watchObservedRunningTime="2026-11-02 17:22:17.438188357 +0000 UTC m=+8.674963026"
Nov 02 17:22:17.676365 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:17.676296    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" podStartSLOduration=7.676279107 podStartE2EDuration="7.676279107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:17.557676063 +0000 UTC m=+8.794450732" watchObservedRunningTime="2026-11-02 17:22:17.676279107 +0000 UTC m=+8.913053777"
Nov 02 17:22:17.677287 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:17.677239    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal" podStartSLOduration=7.677229394 podStartE2EDuration="7.677229394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:17.676271759 +0000 UTC m=+8.913046429" watchObservedRunningTime="2026-11-02 17:22:17.677229394 +0000 UTC m=+8.914004063"
Nov 02 17:22:18.102672 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.101770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:18.102672 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.101841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:18.102672 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.101973    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:18.102672 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.102047    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret podName:6f939783-c4a2-4931-9ab9-07f56e14a7ed nodeName:}" failed. No retries permitted until 2026-11-02 17:22:26.102026806 +0000 UTC m=+17.338801477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret") pod "global-pull-secret-syncer-trcmz" (UID: "6f939783-c4a2-4931-9ab9-07f56e14a7ed") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:18.102672 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.102123    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:18.102672 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.102177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:26.102163232 +0000 UTC m=+17.338937891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:18.202682 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.202569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:18.202992 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.202916    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:18.203072 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.203013    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:18.203072 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.203025    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r455q for pod openshift-network-diagnostics/network-check-target-slw2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:18.203182 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.203091    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q podName:25000ba3-538e-482f-bec4-e882e1053b1f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:26.203070102 +0000 UTC m=+17.439844771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r455q" (UniqueName: "kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q") pod "network-check-target-slw2l" (UID: "25000ba3-538e-482f-bec4-e882e1053b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:18.245821 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.245767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:18.246002 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.245837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:18.246002 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.245945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:18.246002 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.245946    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:18.246178 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.246073    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:18.246178 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:18.246162    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:18.350415 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.340605    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj" event={"ID":"cd70ad9e-6914-4bba-8e93-de02ed571a4f","Type":"ContainerStarted","Data":"e9ed2e813adfe04d79982356e5008ec464f9b775db99db9318f1750936306725"}
Nov 02 17:22:18.350415 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:18.348439    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqt8d" event={"ID":"9867d6a5-3527-4058-80a0-1d8dd1e1590a","Type":"ContainerStarted","Data":"439ae6b6343a92ceb5d412b6392a9aa08ac5f12b39b12e71e7f135340736e7f6"}
Nov 02 17:22:19.515606 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:19.515565    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:19.516745 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:19.516502    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:19.637851 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:19.637789    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nqt8d" podStartSLOduration=3.273621128 podStartE2EDuration="9.637769437s" totalImagesPullingTime="6.364148309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:18.468981031 +0000 UTC m=+9.705755701" watchObservedRunningTime="2026-11-02 17:22:19.637769437 +0000 UTC m=+10.874544107"
Nov 02 17:22:20.246813 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.245931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:20.246813 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:20.246074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:20.246813 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.246496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:20.246813 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:20.246594    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:20.246813 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.246660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:20.246813 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:20.246770    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:20.328501 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.328475    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:22:20.360104 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.360067    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4df8n" event={"ID":"55f98f6e-7d08-4d15-9df2-24d5c205a5d3","Type":"ContainerStarted","Data":"40d13b668f2567c63f61e1e75a0286f0212bf7503f6a47aa4f027da5ade4d08a"}
Nov 02 17:22:20.365743 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.365698    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj" event={"ID":"cd70ad9e-6914-4bba-8e93-de02ed571a4f","Type":"ContainerStarted","Data":"f77fd7f5b70bece2f2b6131998401a4e3e7a390a123ae29e30e4c80c3af5da0c"}
Nov 02 17:22:20.481479 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:20.481423    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4df8n" podStartSLOduration=1.571637166 podStartE2EDuration="10.481408586s" totalImagesPullingTime="8.90977142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:20.480681235 +0000 UTC m=+11.717455905" watchObservedRunningTime="2026-11-02 17:22:20.481408586 +0000 UTC m=+11.718183254"
Nov 02 17:22:21.214912 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:21.214777    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:22:20.328498018Z","UUID":"cabdacbc-b592-44e8-ac1f-105ea8e78a04","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:22:21.217139 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:21.216705    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:22:21.217139 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:21.216743    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:22:21.371320 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:21.371282    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hs745" event={"ID":"c9ac25cf-7824-4fca-8c91-d3c7293e3834","Type":"ContainerStarted","Data":"3aad8c977e50904ec2eccee4973e2e847e6987d525e4af57d9282ade404df088"}
Nov 02 17:22:21.375404 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:21.375349    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerStarted","Data":"3fed9542f4f7eab84b32b14e0e5e7156a5083523fa6353a0b826ed263c8cf35c"}
Nov 02 17:22:21.491408 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:21.491351    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hs745" podStartSLOduration=2.630421816 podStartE2EDuration="11.491338657s" totalImagesPullingTime="8.860916841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:21.490912409 +0000 UTC m=+12.727687077" watchObservedRunningTime="2026-11-02 17:22:21.491338657 +0000 UTC m=+12.728113321"
Nov 02 17:22:22.247149 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.246596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:22.247149 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:22.246774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:22.247149 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.246863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:22.247149 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:22.246945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:22.247149 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.247010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:22.247149 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:22.247102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:22.379883 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.379591    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="3fed9542f4f7eab84b32b14e0e5e7156a5083523fa6353a0b826ed263c8cf35c" exitCode=0
Nov 02 17:22:22.380080 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.379778    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerDied","Data":"3fed9542f4f7eab84b32b14e0e5e7156a5083523fa6353a0b826ed263c8cf35c"}
Nov 02 17:22:22.381954 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.381923    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tw9mq" event={"ID":"efe751b0-e2fe-4576-83de-e9dcb258291b","Type":"ContainerStarted","Data":"3a85411d5c757db14a37105a4d0543b110a737469c451b044c3439334c887b7a"}
Nov 02 17:22:22.386674 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.386648    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj" event={"ID":"cd70ad9e-6914-4bba-8e93-de02ed571a4f","Type":"ContainerStarted","Data":"ab04f6dd8eeef0ce01b886335f37fe159b0ff68e611cdd5c44340bc6c35809dd"}
Nov 02 17:22:22.629196 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.629094    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj" podStartSLOduration=1.628921931 podStartE2EDuration="12.629074497s" totalImagesPullingTime="11.000152566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:22.628478715 +0000 UTC m=+13.865253380" watchObservedRunningTime="2026-11-02 17:22:22.629074497 +0000 UTC m=+13.865849164"
Nov 02 17:22:22.738552 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:22.738491    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tw9mq" podStartSLOduration=3.14231553 podStartE2EDuration="12.738473031s" totalImagesPullingTime="9.596157501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:22.737825885 +0000 UTC m=+13.974600550" watchObservedRunningTime="2026-11-02 17:22:22.738473031 +0000 UTC m=+13.975247695"
Nov 02 17:22:23.390858 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:23.390787    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmr9j" event={"ID":"6a5b0c75-fe80-4472-b423-97785579f253","Type":"ContainerStarted","Data":"520d74ca21688a04df43c417eba0eff451ab84455b5dfa9cc505b02a71a8c79a"}
Nov 02 17:22:23.502745 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:23.502700    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pmr9j" podStartSLOduration=2.090430691 podStartE2EDuration="13.502685854s" totalImagesPullingTime="11.412255163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:23.502561676 +0000 UTC m=+14.739336342" watchObservedRunningTime="2026-11-02 17:22:23.502685854 +0000 UTC m=+14.739460516"
Nov 02 17:22:24.246191 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:24.246161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:24.246191 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:24.246188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:24.246394 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:24.246197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:24.246394 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:24.246271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:24.246394 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:24.246345    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:24.246531 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:24.246415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:24.395738 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:24.395703    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="f6a671bf3f6790c04061372789f370d9d44ad578f86608a1d26c31b102ff9b9a" exitCode=0
Nov 02 17:22:24.396358 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:24.395795    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerDied","Data":"f6a671bf3f6790c04061372789f370d9d44ad578f86608a1d26c31b102ff9b9a"}
Nov 02 17:22:26.170965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.170928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:26.170965 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.170985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:26.171676 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.171117    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:26.171676 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.171127    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:26.171676 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.171201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.171180877 +0000 UTC m=+33.407955539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:22:26.171676 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.171223    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret podName:6f939783-c4a2-4931-9ab9-07f56e14a7ed nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.171212862 +0000 UTC m=+33.407987519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret") pod "global-pull-secret-syncer-trcmz" (UID: "6f939783-c4a2-4931-9ab9-07f56e14a7ed") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:22:26.246360 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.246314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:26.246516 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.246323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:26.246516 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.246446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:26.246516 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.246479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:26.246660 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.246566    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:26.246706 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.246690    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:26.271642 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.271588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:26.271812 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.271769    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:22:26.271812 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.271789    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:22:26.271812 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.271799    2577 projected.go:196] Error preparing data for projected volume kube-api-access-r455q for pod openshift-network-diagnostics/network-check-target-slw2l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:26.271921 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:26.271849    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q podName:25000ba3-538e-482f-bec4-e882e1053b1f nodeName:}" failed. No retries permitted until 2026-11-02 17:22:42.271836159 +0000 UTC m=+33.508610803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r455q" (UniqueName: "kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q") pod "network-check-target-slw2l" (UID: "25000ba3-538e-482f-bec4-e882e1053b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:22:26.402955 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.402914    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="48a65da30606e58bd584debbf7c277ba20a683fa45fbf9cb1f9028844c983598" exitCode=0
Nov 02 17:22:26.403137 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:26.402986    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerDied","Data":"48a65da30606e58bd584debbf7c277ba20a683fa45fbf9cb1f9028844c983598"}
Nov 02 17:22:28.246393 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.246169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:28.246930 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.246272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:28.246930 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:28.246501    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:28.246930 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.246286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:28.246930 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:28.246587    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:28.246930 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:28.246647    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:28.411499 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.411463    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"d648f67f3bd804822877e9149069144f5b64bbb53f465b9bdbd3b81b7056dd63"}
Nov 02 17:22:28.411609 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.411562    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"6a240266871b327f4e9629ba6b7f4967ebf49ebcee03298463162aede6258bf7"}
Nov 02 17:22:28.411689 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.411612    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"91efab0ff8da75a94093bf2c72bcd80347e85fb1954536ae38162e71a035b9d8"}
Nov 02 17:22:28.411689 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.411684    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"f2f321b951c3f9d6d43a299c0cbfe56c6972cdd5e3810a7e2941ed85613d4c7d"}
Nov 02 17:22:28.411781 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.411731    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"18bb83c00cba980ad98a5a2959d5fdd84a611910dfa536325d7922316e143a92"}
Nov 02 17:22:28.411781 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:28.411776    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"3d139bb49a1a25dff47c83a966facec135acb2706f855fe0b4b294caa60e7e96"}
Nov 02 17:22:29.415582 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:29.415549    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="92c7add7d1ee37683687e8187601e6f6814228a3eff53cded7a17afff27d6f6f" exitCode=0
Nov 02 17:22:29.416006 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:29.415603    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerDied","Data":"92c7add7d1ee37683687e8187601e6f6814228a3eff53cded7a17afff27d6f6f"}
Nov 02 17:22:29.495012 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:29.494984    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:29.495135 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:29.495122    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:22:29.495608 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:29.495588    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:30.246492 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:30.246401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:30.246661 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:30.246491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:30.246661 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:30.246587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:30.246661 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:30.246582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:30.246766 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:30.246727    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:30.246815 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:30.246797    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:30.421043 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:30.421011    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"079a2de485cbf0dffe8b1b6277a8a03ca61ff559942f5122028af4298e22e127"}
Nov 02 17:22:30.423771 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:30.423745    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="103cbcc3b0d755296179cc55e4235112f2adbbf623d35c9f8acb9a74031e5722" exitCode=0
Nov 02 17:22:30.423906 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:30.423827    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerDied","Data":"103cbcc3b0d755296179cc55e4235112f2adbbf623d35c9f8acb9a74031e5722"}
Nov 02 17:22:31.428896 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:31.428856    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9spt5" event={"ID":"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f","Type":"ContainerStarted","Data":"bf37323f706818413e2b974aa54a17034ac8fb95b262dbe4fd93ce6d403abc07"}
Nov 02 17:22:31.641742 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:31.641686    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9spt5" podStartSLOduration=8.85933307 podStartE2EDuration="21.641672757s" totalImagesPullingTime="12.782339687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:31.591453641 +0000 UTC m=+22.828228305" watchObservedRunningTime="2026-11-02 17:22:31.641672757 +0000 UTC m=+22.878447421"
Nov 02 17:22:32.245873 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:32.245790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:32.245873 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:32.245821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:32.245873 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:32.245835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:32.246071 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:32.245921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:32.246071 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:32.246036    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:32.246139 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:32.246121    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:32.435139 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:32.435108    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" event={"ID":"29109340-d347-45c7-a6dc-891cc6f752a9","Type":"ContainerStarted","Data":"f6167bfcd243edf52b4ee72b23a2518bef2672cf9273e70f659a86b02dddc3e6"}
Nov 02 17:22:32.637967 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:32.637872    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c" podStartSLOduration=6.044560859 podStartE2EDuration="22.637859671s" totalImagesPullingTime="16.593298812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:32.637360489 +0000 UTC m=+23.874135150" watchObservedRunningTime="2026-11-02 17:22:32.637859671 +0000 UTC m=+23.874634336"
Nov 02 17:22:33.437974 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:33.437946    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:33.438464 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:33.438023    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:33.438464 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:33.438072    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:33.454043 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:33.454016    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:33.454593 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:33.454568    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:34.246259 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.246220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:34.246259 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.246249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:34.246487 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:34.246348    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:34.246487 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:34.246455    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:34.246603 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.246504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:34.246679 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:34.246615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:34.768997 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.768700    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2b8zt"]
Nov 02 17:22:34.769490 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.769235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:34.769490 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.769342    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-trcmz"]
Nov 02 17:22:34.769490 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:34.769363    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:34.769490 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.769453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:34.769714 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:34.769549    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:34.770056 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.770030    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slw2l"]
Nov 02 17:22:34.770157 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:34.770109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:34.770216 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:34.770197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:36.246374 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:36.246289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:36.246729 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:36.246305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:36.246729 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:36.246418    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:36.246729 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:36.246427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:36.246729 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:36.246499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:36.246729 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:36.246582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:38.246046 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:38.245945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:38.246449 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:38.246066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:38.246449 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:38.246070    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slw2l" podUID="25000ba3-538e-482f-bec4-e882e1053b1f"
Nov 02 17:22:38.246449 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:38.246176    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-trcmz" podUID="6f939783-c4a2-4931-9ab9-07f56e14a7ed"
Nov 02 17:22:38.246449 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:38.246075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:38.246449 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:38.246288    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2b8zt" podUID="00939ab6-259b-42cd-be51-95182c9ba479"
Nov 02 17:22:38.596146 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:38.596064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:38.903791 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:38.903698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:39.555396 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.555360    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeReady"
Nov 02 17:22:39.555873 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.555499    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 17:22:39.709635 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.709602    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n8b9d"]
Nov 02 17:22:39.712730 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.712712    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rxx72"]
Nov 02 17:22:39.712850 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.712828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.715795 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.715773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:39.719683 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.719660    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8b9d"]
Nov 02 17:22:39.723201 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.723134    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rxx72"]
Nov 02 17:22:39.765040 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.765008    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:22:39.765040 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.765024    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:22:39.765040 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.765067    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b28rw\""
Nov 02 17:22:39.769147 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.769126    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:22:39.769398 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.769381    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:22:39.865812 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.865776    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:22:39.865972 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.865900    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svxv2\""
Nov 02 17:22:39.867072 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.867056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.867117 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.867085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19284cc9-6243-40af-862b-dc011abb8b7c-config-volume\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.867117 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.867106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hqt\" (UniqueName: \"kubernetes.io/projected/1e6b3bff-315b-4fd5-b428-a2da329b0cea-kube-api-access-67hqt\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:39.867205 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.867134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfs8\" (UniqueName: \"kubernetes.io/projected/19284cc9-6243-40af-862b-dc011abb8b7c-kube-api-access-jtfs8\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.867284 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.867267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19284cc9-6243-40af-862b-dc011abb8b7c-tmp-dir\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.867332 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.867299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e6b3bff-315b-4fd5-b428-a2da329b0cea-cert\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:39.968106 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19284cc9-6243-40af-862b-dc011abb8b7c-tmp-dir\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.968106 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e6b3bff-315b-4fd5-b428-a2da329b0cea-cert\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:39.968311 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.968311 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19284cc9-6243-40af-862b-dc011abb8b7c-config-volume\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.968311 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hqt\" (UniqueName: \"kubernetes.io/projected/1e6b3bff-315b-4fd5-b428-a2da329b0cea-kube-api-access-67hqt\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:39.968311 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfs8\" (UniqueName: \"kubernetes.io/projected/19284cc9-6243-40af-862b-dc011abb8b7c-kube-api-access-jtfs8\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.968311 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:39.968299    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:39.968568 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:39.968367    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:40.468352923 +0000 UTC m=+31.705127566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:39.968568 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19284cc9-6243-40af-862b-dc011abb8b7c-tmp-dir\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.968917 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.968894    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19284cc9-6243-40af-862b-dc011abb8b7c-config-volume\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:39.972716 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:39.972701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e6b3bff-315b-4fd5-b428-a2da329b0cea-cert\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:40.035839 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.035663    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfs8\" (UniqueName: \"kubernetes.io/projected/19284cc9-6243-40af-862b-dc011abb8b7c-kube-api-access-jtfs8\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:40.051199 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.051174    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hqt\" (UniqueName: \"kubernetes.io/projected/1e6b3bff-315b-4fd5-b428-a2da329b0cea-kube-api-access-67hqt\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:40.246252 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.246174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:40.246398 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.246301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:40.246398 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.246189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:40.299076 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299039    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:22:40.299076 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299072    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:22:40.299265 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299141    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:22:40.299265 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299189    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:22:40.299265 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299223    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\""
Nov 02 17:22:40.299265 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299229    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\""
Nov 02 17:22:40.299383 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.299270    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\""
Nov 02 17:22:40.333839 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.333806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:40.472781 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.472742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:40.472926 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:40.472909    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:40.472991 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:40.472978    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:41.472962039 +0000 UTC m=+32.709736681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:40.719972 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.719890    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rxx72"]
Nov 02 17:22:40.726077 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:40.726050    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6b3bff_315b_4fd5_b428_a2da329b0cea.slice/crio-4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964 WatchSource:0}: Error finding container 4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964: Status 404 returned error can't find the container with id 4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964
Nov 02 17:22:40.867108 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:40.867074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:41.129418 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:41.129331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:41.457602 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:41.457512    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rxx72" event={"ID":"1e6b3bff-315b-4fd5-b428-a2da329b0cea","Type":"ContainerStarted","Data":"4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964"}
Nov 02 17:22:41.481304 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:41.481271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:41.481436 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:41.481420    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:41.481488 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:41.481481    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:43.481466411 +0000 UTC m=+34.718241055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:42.186951 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.186910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:42.187741 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.186998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:42.187741 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:42.187064    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:42.187741 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:42.187143    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:23:14.187121827 +0000 UTC m=+65.423896485 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : secret "metrics-daemon-secret" not found
Nov 02 17:22:42.189883 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.189861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:42.287741 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.287703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:42.290600 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.290575    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:42.361555 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.361523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:42.365708 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.365678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:42.758607 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.758529    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slw2l"]
Nov 02 17:22:42.764411 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:42.764378    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25000ba3_538e_482f_bec4_e882e1053b1f.slice/crio-9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae WatchSource:0}: Error finding container 9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae: Status 404 returned error can't find the container with id 9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae
Nov 02 17:22:42.808380 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:42.808349    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-trcmz"]
Nov 02 17:22:42.813701 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:42.813572    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f939783_c4a2_4931_9ab9_07f56e14a7ed.slice/crio-ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18 WatchSource:0}: Error finding container ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18: Status 404 returned error can't find the container with id ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18
Nov 02 17:22:43.020949 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:43.020813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:43.217785 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:43.217740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:43.468117 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:43.468025    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-trcmz" event={"ID":"6f939783-c4a2-4931-9ab9-07f56e14a7ed","Type":"ContainerStarted","Data":"ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18"}
Nov 02 17:22:43.469200 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:43.469166    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slw2l" event={"ID":"25000ba3-538e-482f-bec4-e882e1053b1f","Type":"ContainerStarted","Data":"9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae"}
Nov 02 17:22:43.498693 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:43.498655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:43.498869 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:43.498818    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:43.498916 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:43.498889    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:47.498872326 +0000 UTC m=+38.735646985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:45.385283 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.385241    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pmnsd"]
Nov 02 17:22:45.420502 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.420463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.448988 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.448951    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pmnsd"]
Nov 02 17:22:45.472591 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.472505    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:22:45.477743 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.477706    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rxx72" event={"ID":"1e6b3bff-315b-4fd5-b428-a2da329b0cea","Type":"ContainerStarted","Data":"603dea4ab2fc0765db9df9b7a12d78db4ea4712190b43c85f1d01dbf069622a8"}
Nov 02 17:22:45.497597 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.496989    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:22:45.497597 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.496989    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:22:45.497597 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.497547    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:22:45.497597 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.497595    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2zcw\""
Nov 02 17:22:45.513902 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.513861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.514085 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.513918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.514085 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.513943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-crio-socket\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.514085 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.513965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-data-volume\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.514085 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.514046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpf4\" (UniqueName: \"kubernetes.io/projected/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-api-access-bfpf4\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615024 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.614981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpf4\" (UniqueName: \"kubernetes.io/projected/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-api-access-bfpf4\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615200 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.615056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615200 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.615092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615200 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.615112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-crio-socket\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615200 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.615135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-data-volume\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615815 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:45.615795    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:45.615919 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:45.615866    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.115850235 +0000 UTC m=+37.352624895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:45.615988 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:45.615942    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:45.615988 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.615943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-crio-socket\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.615988 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:45.615969    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:46.115961926 +0000 UTC m=+37.352736583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:45.616144 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.616070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-data-volume\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.658717 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.657718    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rxx72" podStartSLOduration=2.177865217 podStartE2EDuration="6.657687965s" totalImagesPullingTime="4.479822748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:45.657503326 +0000 UTC m=+36.894277990" watchObservedRunningTime="2026-11-02 17:22:45.657687965 +0000 UTC m=+36.894462629"
Nov 02 17:22:45.678127 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.677777    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kxkns-mjrds"]
Nov 02 17:22:45.681777 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.681737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpf4\" (UniqueName: \"kubernetes.io/projected/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-api-access-bfpf4\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:45.693359 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.693327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.701165 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.701134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:45.735818 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.735695    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kxkns-mjrds"]
Nov 02 17:22:45.745508 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.745466    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gfr4c\""
Nov 02 17:22:45.745683 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.745530    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:22:45.796011 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.795981    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:22:45.817311 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.817280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.817486 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.817363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.817486 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.817434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.817486 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.817481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.918612 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.918578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.918827 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.918660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.918827 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.918708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.918827 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.918725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.919190 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.919153    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.919278 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.919222    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.919497 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.919480    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:45.977407 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:45.977374    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:46.022279 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.022197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:46.120621 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.120559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:46.120942 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.120885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:46.121270 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:46.121155    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:46.122022 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:46.121989    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:47.121230603 +0000 UTC m=+38.358005250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:46.122846 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:46.122431    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:46.122846 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:46.122503    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:47.122485153 +0000 UTC m=+38.359259814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:46.327921 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.327828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:46.423309 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.423276    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kxkns-mjrds"]
Nov 02 17:22:46.428910 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:46.428878    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1447be_4ab8_40c9_8606_de70d0920e65.slice/crio-33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c WatchSource:0}: Error finding container 33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c: Status 404 returned error can't find the container with id 33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c
Nov 02 17:22:46.459231 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.459200    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"]
Nov 02 17:22:46.481363 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.481337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:46.481913 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.481872    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kxkns-mjrds" event={"ID":"df1447be-4ab8-40c9-8606-de70d0920e65","Type":"ContainerStarted","Data":"33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c"}
Nov 02 17:22:46.521446 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.521412    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"]
Nov 02 17:22:46.534219 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.534190    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xdlgk\""
Nov 02 17:22:46.548192 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.548162    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:46.548385 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.548212    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:22:46.566453 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.566418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:46.625273 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.625180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv87\" (UniqueName: \"kubernetes.io/projected/7d3b166c-1f4f-4645-bc93-d76093c412e1-kube-api-access-glv87\") pod \"migrator-66df5d496b-jjlxv\" (UID: \"7d3b166c-1f4f-4645-bc93-d76093c412e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:46.725769 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.725731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv87\" (UniqueName: \"kubernetes.io/projected/7d3b166c-1f4f-4645-bc93-d76093c412e1-kube-api-access-glv87\") pod \"migrator-66df5d496b-jjlxv\" (UID: \"7d3b166c-1f4f-4645-bc93-d76093c412e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:46.764493 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.764450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:46.782920 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.782885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv87\" (UniqueName: \"kubernetes.io/projected/7d3b166c-1f4f-4645-bc93-d76093c412e1-kube-api-access-glv87\") pod \"migrator-66df5d496b-jjlxv\" (UID: \"7d3b166c-1f4f-4645-bc93-d76093c412e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:46.807789 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.807746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:46.971111 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:46.971059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:47.130061 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.130028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:47.130250 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.130068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:47.130250 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:47.130151    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:47.130250 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:47.130199    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:47.130250 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:47.130226    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:49.130211869 +0000 UTC m=+40.366986532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:47.130391 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:47.130267    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:49.130248412 +0000 UTC m=+40.367023056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:47.243454 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.243361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:47.310233 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.310197    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"]
Nov 02 17:22:47.314515 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:47.314469    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3b166c_1f4f_4645_bc93_d76093c412e1.slice/crio-781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25 WatchSource:0}: Error finding container 781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25: Status 404 returned error can't find the container with id 781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25
Nov 02 17:22:47.446890 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.446836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:47.486599 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.486565    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv" event={"ID":"7d3b166c-1f4f-4645-bc93-d76093c412e1","Type":"ContainerStarted","Data":"781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25"}
Nov 02 17:22:47.488689 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.488662    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slw2l" event={"ID":"25000ba3-538e-482f-bec4-e882e1053b1f","Type":"ContainerStarted","Data":"fb3897a82414cb45e276be287e4ed69fc82f872174d2bd977262d27e2cbfe7bd"}
Nov 02 17:22:47.488834 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.488814    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:47.533723 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.533618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:47.533889 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:47.533766    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:47.533889 ip-10-0-136-192 kubenswrapper[2577]: E1102 17:22:47.533831    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:55.533818439 +0000 UTC m=+46.770593081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:47.964182 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.964069    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-slw2l" podStartSLOduration=33.779219149 podStartE2EDuration="37.964048054s" totalImagesPullingTime="4.184828905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:47.605918067 +0000 UTC m=+38.842692732" watchObservedRunningTime="2026-11-02 17:22:47.964048054 +0000 UTC m=+39.200822719"
Nov 02 17:22:47.964440 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.964416    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tf45l"]
Nov 02 17:22:47.995552 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:47.995510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.023778 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.023751    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tf45l"]
Nov 02 17:22:48.051404 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.051347    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 17:22:48.051554 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.051471    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 17:22:48.051554 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.051506    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 17:22:48.051554 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.051534    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 17:22:48.051759 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.051729    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wfdxg\""
Nov 02 17:22:48.139190 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.139153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a793afe1-5ad9-4746-8a70-631f91206420-signing-key\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.139377 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.139210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6f7h\" (UniqueName: \"kubernetes.io/projected/a793afe1-5ad9-4746-8a70-631f91206420-kube-api-access-k6f7h\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.139377 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.139332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a793afe1-5ad9-4746-8a70-631f91206420-signing-cabundle\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.239912 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.239840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a793afe1-5ad9-4746-8a70-631f91206420-signing-key\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.239912 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.239885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6f7h\" (UniqueName: \"kubernetes.io/projected/a793afe1-5ad9-4746-8a70-631f91206420-kube-api-access-k6f7h\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.240073 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.239935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a793afe1-5ad9-4746-8a70-631f91206420-signing-cabundle\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.240461 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.240441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a793afe1-5ad9-4746-8a70-631f91206420-signing-cabundle\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.242310 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.242285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a793afe1-5ad9-4746-8a70-631f91206420-signing-key\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.303942 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.303909    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6f7h\" (UniqueName: \"kubernetes.io/projected/a793afe1-5ad9-4746-8a70-631f91206420-kube-api-access-k6f7h\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.320297 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.320264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:48.391606 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.391562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:48.678611 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.678578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:48.719819 ip-10-0-136-192 kubenswrapper[2577]: I1102 17:22:48.719788    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tf45l"]
Nov 02 17:22:48.725571 ip-10-0-136-192 kubenswrapper[2577]: W1102 17:22:48.725542    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda793afe1_5ad9_4746_8a70_631f91206420.slice/crio-fd79e8c8548809f704ba833a4acc10e71e6d07db4edbaaf83975e60c588316d6 WatchSource:0}: Error finding container fd79e8c8548809f704ba833a4acc10e71e6d07db4edbaaf83975e60c588316d6: Status 404 returned error can't find the container with id fd79e8c8548809f704ba833a4acc10e71e6d07db4edbaaf83975e60c588316d6
Nov 02 17:22:49.090267 ip-10-0-136-192 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 17:22:49.109332 ip-10-0-136-192 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 17:22:49.109597 ip-10-0-136-192 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 17:22:49.109687 ip-10-0-136-192 systemd[1]: kubelet.service: Consumed 2.533s CPU time.
Nov 02 17:22:49.128343 ip-10-0-136-192 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:22:49.223227 ip-10-0-136-192 kubenswrapper[4476]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:49.223227 ip-10-0-136-192 kubenswrapper[4476]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:22:49.223227 ip-10-0-136-192 kubenswrapper[4476]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:49.223227 ip-10-0-136-192 kubenswrapper[4476]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:22:49.225266 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225182    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.225266 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225267    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.225266 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225270    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225274    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225277    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225280    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225282    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225285    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225287    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225290    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225292    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225295    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225298    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225300    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225303    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225305    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225308    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225310    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225312    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225315    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225318    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225320    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.225370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225323    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225325    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225328    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225332    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225336    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225345    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225348    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225351    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225353    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225356    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225359    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225362    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225365    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225368    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225370    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225373    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225376    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225378    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225381    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.225873 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225384    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225386    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225388    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225391    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225393    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225396    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225399    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225401    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225404    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225407    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225410    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225415    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225418    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225420    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225423    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225425    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225429    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225433    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225436    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.226337 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225440    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225443    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225445    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225448    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225451    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225453    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225456    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225458    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225461    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225463    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225466    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225468    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225471    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225473    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225476    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225478    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225481    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225484    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225487    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225490    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.226825 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225492    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225495    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225497    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225500    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225504    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225506    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225509    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225511    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225514    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225516    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225519    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225521    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225524    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225528    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225531    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225533    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225536    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225538    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225540    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225543    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.227311 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225545    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225548    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225550    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225553    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225555    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.225558    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226356    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226364    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226367    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226370    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226373    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226376    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226379    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226381    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226384    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226386    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226390    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226393    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226395    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.227816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226398    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226400    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226403    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226405    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226408    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226411    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226413    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226416    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226419    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226422    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226424    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226427    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226429    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226431    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226434    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226436    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226439    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226441    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226443    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226446    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.228294 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226448    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226451    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226454    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226456    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226459    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226461    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226464    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226468    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226472    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226475    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226478    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226481    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226483    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226486    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226488    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226491    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226493    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226495    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226498    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.228788 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226500    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226504    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226507    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226509    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226512    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226514    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226517    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226519    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226521    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226524    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226527    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226529    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226531    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226534    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226536    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226539    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226542    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226544    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226547    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226550    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.229256 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226552    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226555    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226557    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226560    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226563    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226565    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226568    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226570    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226572    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226575    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226577    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226580    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226582    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226585    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226588    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226591    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226593    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226596    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226598    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.229800 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226600    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226603    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226605    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226608    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226612    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226616    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226619    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226622    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226644    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226648    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226656    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226661    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226664    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226666    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.226669    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227215    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227221    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227224    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227227    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.230263 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227230    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227232    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227234    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227237    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227239    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227242    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227244    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227247    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227250    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227253    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227255    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227258    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227261    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227263    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227266    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227268    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227270    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227273    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227275    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.230742 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227278    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227280    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227285    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227298    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227303    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227311    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227315    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227318    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227321    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227324    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227326    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227329    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227331    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227334    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227336    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227339    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227341    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227346    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227349    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.231288 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227352    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227354    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227357    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227360    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227363    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227365    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227368    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227370    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227373    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227375    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227378    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227380    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227382    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227385    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227387    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227391    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227393    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227397    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227399    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.232144 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227402    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227405    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227407    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227410    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227412    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227414    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227417    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227419    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227421    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227424    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227427    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227429    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227431    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227434    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227436    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227438    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227441    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227444    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227446    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.232972 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227449    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227451    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227454    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227456    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227458    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227461    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227463    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227466    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227468    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227472    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227474    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227480    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227484    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227487    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227490    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227492    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227495    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227497    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227500    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227502    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.233811 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227505    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227507    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227509    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227511    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227514    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.227517    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227609    4476 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227617    4476 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227637    4476 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227642    4476 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227647    4476 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227652    4476 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227656    4476 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227660    4476 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227664    4476 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227667    4476 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227670    4476 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227673    4476 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227676    4476 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227679    4476 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227682    4476 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227687    4476 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227690    4476 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:22:49.234747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227694    4476 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227699    4476 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227702    4476 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227705    4476 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227708    4476 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227711    4476 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227718    4476 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227721    4476 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227725    4476 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227728    4476 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227731    4476 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227734    4476 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227737    4476 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227740    4476 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227743    4476 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227747    4476 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227750    4476 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227753    4476 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227757    4476 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227760    4476 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227763    4476 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227768    4476 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227771    4476 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227774    4476 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:22:49.235751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227777    4476 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227780    4476 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227785    4476 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227787    4476 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227790    4476 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227793    4476 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227798    4476 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227801    4476 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227806    4476 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227809    4476 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227812    4476 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227815    4476 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227818    4476 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227823    4476 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227826    4476 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227830    4476 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227833    4476 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227836    4476 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227839    4476 flags.go:64] FLAG: --help="false"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227842    4476 flags.go:64] FLAG: --hostname-override="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227845    4476 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227848    4476 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227851    4476 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:22:49.236508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227855    4476 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227858    4476 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227861    4476 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227864    4476 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227867    4476 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227869    4476 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227872    4476 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227884    4476 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227887    4476 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227890    4476 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227893    4476 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227896    4476 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227899    4476 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227902    4476 flags.go:64] FLAG: --lock-file=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227905    4476 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227910    4476 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227913    4476 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227919    4476 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227922    4476 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227925    4476 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227929    4476 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227932    4476 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227935    4476 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227938    4476 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227941    4476 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:22:49.237093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227945    4476 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227948    4476 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227952    4476 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227955    4476 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227958    4476 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227962    4476 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227965    4476 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227967    4476 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227970    4476 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227973    4476 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227980    4476 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227983    4476 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227986    4476 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227989    4476 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227992    4476 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227996    4476 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.227999    4476 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228002    4476 flags.go:64] FLAG: --port="10250"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228005    4476 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228007    4476 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-019afcce8afbd8359"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228011    4476 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228014    4476 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228018    4476 flags.go:64] FLAG: --register-node="true"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228021    4476 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228026    4476 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228029    4476 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:22:49.237714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228032    4476 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228035    4476 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228039    4476 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228042    4476 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228045    4476 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228048    4476 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228051    4476 flags.go:64] FLAG: --runonce="false"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228054    4476 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228057    4476 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228060    4476 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228062    4476 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228065    4476 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228068    4476 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228071    4476 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228074    4476 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228078    4476 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228081    4476 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228083    4476 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228086    4476 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228089    4476 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228092    4476 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228095    4476 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228101    4476 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228104    4476 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228107    4476 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:22:49.238489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228111    4476 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228114    4476 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228117    4476 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228122    4476 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228124    4476 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228129    4476 flags.go:64] FLAG: --v="2"
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228133    4476 flags.go:64] FLAG: --version="false"
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228137    4476 flags.go:64] FLAG: --vmodule=""
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228141    4476 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228144    4476 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228286    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228290    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228292    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228295    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228298    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228301    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228304    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228307    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228310    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228312    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228315    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.239202 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228318    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228320    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228323    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228325    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228328    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228330    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228333    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228335    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228338    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228341    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228343    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228346    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228348    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228350    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228355    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228357    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228361    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228364    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228366    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228368    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.239778 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228371    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228373    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228376    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228380    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228383    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228386    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228388    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228391    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228393    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228396    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228398    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228401    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228403    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228406    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228408    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228411    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228413    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228416    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228418    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228421    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.240373 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228423    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228425    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228428    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228431    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228434    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228436    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228439    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228442    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228446    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228449    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228451    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228454    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228456    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228459    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228461    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228464    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228466    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228469    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228472    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228474    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.240930 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228477    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228479    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228482    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228484    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228487    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228489    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228493    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228496    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228499    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228501    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228504    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228506    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228509    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228511    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228513    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228516    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228519    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228521    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228525    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.241456 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228528    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228531    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228535    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228539    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228542    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228544    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228547    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228550    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228552    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228555    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228557    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228559    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228562    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228564    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228567    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.241940 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228569    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.228595    4476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228658    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228664    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228666    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228669    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228672    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228674    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228677    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228679    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228682    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228684    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228687    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228689    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228692    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.242297 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228695    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228698    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228704    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228708    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228711    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228713    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228716    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228718    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228721    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228723    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228726    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228728    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228731    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228734    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228736    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228738    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228741    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228744    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228746    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.242727 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228748    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228750    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228753    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228755    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228758    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228760    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228763    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228765    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228768    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228770    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228772    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228775    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228777    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228781    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228785    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228789    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228793    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228796    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228799    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228801    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.243180 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228804    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228807    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228809    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228811    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228814    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228816    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228819    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228821    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228824    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228826    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228829    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228831    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228834    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228836    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228839    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228841    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228843    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228846    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228848    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228851    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.243680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228853    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228856    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228858    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228860    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228863    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228866    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228869    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228872    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228876    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228878    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228881    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228883    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228885    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228888    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228890    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228893    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228895    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228898    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228900    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228903    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.244153 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228905    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228907    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228910    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228912    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228915    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228917    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228919    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228922    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228924    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228927    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228929    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228932    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228936    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.228940    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.233731    4476 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.233750    4476 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.233770    4476 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.233777    4476 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233883    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.244652 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233891    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233896    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233914    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233919    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233923    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233928    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233932    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233937    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233941    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233945    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233950    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233954    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233958    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233963    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233967    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233972    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233977    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233981    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233985    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.245131 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233989    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233993    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.233998    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234002    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234006    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234011    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234016    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234020    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234024    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234029    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234033    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234037    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234041    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234046    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234051    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234055    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234059    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234064    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234068    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.245760 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234072    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234076    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234080    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234085    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234089    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234094    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234098    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234103    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234107    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234111    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234115    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234120    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234126    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234132    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234137    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234141    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234146    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234151    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234157    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.246421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234162    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234167    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234172    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234176    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234180    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234184    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234189    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234193    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234198    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234202    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234206    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234211    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234215    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234219    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234223    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234227    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234232    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234236    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234241    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234245    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.247110 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234250    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234255    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234259    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234263    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234267    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234271    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234275    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234280    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234284    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234288    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234293    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234297    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234301    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234306    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234310    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234314    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234319    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234323    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234327    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234332    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.247595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234336    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234341    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234345    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234350    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234354    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234358    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234363    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234367    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.234407    4476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234853    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234870    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234876    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234881    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234894    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234900    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234906    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.248108 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234911    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234921    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234925    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234932    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234938    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234942    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234950    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234954    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234959    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234969    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234973    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234978    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234983    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234988    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234992    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.234997    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235001    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235005    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235012    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.248504 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235016    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235021    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235025    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235035    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235040    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235044    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235049    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235054    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235058    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235065    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235072    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235078    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235082    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235087    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235092    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235101    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235106    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235111    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235115    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235120    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.248983 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235124    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235129    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235133    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235138    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235142    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235146    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235150    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235155    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235164    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235168    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235173    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235178    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235183    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235188    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235192    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235196    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235201    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235205    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235210    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.249466 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235214    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235219    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235228    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235232    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235237    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235242    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235246    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235250    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235255    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235259    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235263    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235268    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235272    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235277    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235329    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235336    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235340    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235344    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235349    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.249953 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235354    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235360    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235367    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235373    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235377    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235382    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235386    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235391    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235401    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235405    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235410    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235415    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235419    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235424    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235428    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235433    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235437    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235442    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235446    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.250435 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235450    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235460    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235464    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.235519    4476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235708    4476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235717    4476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235723    4476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235727    4476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235888    4476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235908    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235912    4476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235915    4476 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235918    4476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235921    4476 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235924    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235926    4476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:22:49.250961 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235929    4476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235931    4476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235934    4476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235938    4476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235940    4476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235945    4476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235950    4476 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235955    4476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235958    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235961    4476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235964    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235968    4476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235971    4476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235974    4476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235977    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235979    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235982    4476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235985    4476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235987    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:22:49.251593 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235990    4476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235992    4476 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235995    4476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.235998    4476 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236001    4476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236004    4476 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236006    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236009    4476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236012    4476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236014    4476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236017    4476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236020    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236022    4476 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236025    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236028    4476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236030    4476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236033    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236035    4476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236040    4476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236043    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:22:49.252408 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236045    4476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236048    4476 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236051    4476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236054    4476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236057    4476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236060    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236063    4476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236065    4476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236068    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236070    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236073    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236076    4476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236078    4476 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236081    4476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236083    4476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236089    4476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236091    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236094    4476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236097    4476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236099    4476 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:22:49.253023 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236102    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236105    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236109    4476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236112    4476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236115    4476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236118    4476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236121    4476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236124    4476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236126    4476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236129    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236132    4476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236134    4476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236137    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236140    4476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236142    4476 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236145    4476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236147    4476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236152    4476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236154    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:22:49.253529 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236157    4476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236159    4476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236162    4476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236164    4476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236169    4476 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236171    4476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236174    4476 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236176    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236180    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236183    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236186    4476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236189    4476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236191    4476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236194    4476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236196    4476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:49.236199    4476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.236273    4476 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237244    4476 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237304    4476 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 17:22:49.254093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237481    4476 server.go:1014] "Starting client certificate rotation"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237643    4476 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237693    4476 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:17:10 +0000 UTC" deadline="2028-05-07 12:29:08.804483708 +0000 UTC"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237725    4476 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13243h6m19.566762282s"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.237930    4476 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.239045    4476 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.240774    4476 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.242214    4476 log.go:25] "Validated CRI v1 image API"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.242947    4476 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:22:49.254925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.244516    4476 fs.go:135] Filesystem UUIDs: map[076be6ac-4f06-4e2f-a697-40e47d3cb75f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8c8e9bc4-b02e-4ed7-af65-a9cf9fd25127:/dev/nvme0n1p3]
Nov 02 17:22:49.255226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.244533    4476 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd79e8c8548809f704ba833a4acc10e71e6d07db4edbaaf83975e60c588316d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd79e8c8548809f704ba833a4acc10e71e6d07db4edbaaf83975e60c588316d6/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00939ab6-259b-42cd-be51-95182c9ba479/volumes/kubernetes.io~projected/kube-api-access-hppnh:{mountpoint:/var/lib/kubelet/pods/00939ab6-259b-42cd-be51-95182c9ba479/volumes/kubernetes.io~projected/kube-api-access-hppnh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f/volumes/kubernetes.io~projected/kube-api-access-mcsqh:{mountpoint:/var/lib/kubelet/pods/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f/volumes/kubernetes.io~projected/kube-api-access-mcsqh major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19284cc9-6243-40af-862b-dc011abb8b7c/volumes/kubernetes.io~projected/kube-api-access-jtfs8:{mountpoint:/var/lib/kubelet/pods/19284cc9-6243-40af-862b-dc011abb8b7c/volumes/kubernetes.io~projected/kube-api-access-jtfs8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e6b3bff-315b-4fd5-b428-a2da329b0cea/volumes/kubernetes.io~projected/kube-api-access-67hqt:{mountpoint:/var/lib/kubelet/pods/1e6b3bff-315b-4fd5-b428-a2da329b0cea/volumes/kubernetes.io~projected/kube-api-access-67hqt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e6b3bff-315b-4fd5-b428-a2da329b0cea/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1e6b3bff-315b-4fd5-b428-a2da329b0cea/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25000ba3-538e-482f-bec4-e882e1053b1f/volumes/kubernetes.io~projected/kube-api-access-r455q:{mountpoint:/var/lib/kubelet/pods/25000ba3-538e-482f-bec4-e882e1053b1f/volumes/kubernetes.io~projected/kube-api-access-r455q major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volumes/kubernetes.io~projected/kube-api-access-9f548:{mountpoint:/var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volumes/kubernetes.io~projected/kube-api-access-9f548 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f80bf07-9f23-46af-b53f-35bbd3157b6d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3f80bf07-9f23-46af-b53f-35bbd3157b6d/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f98f6e-7d08-4d15-9df2-24d5c205a5d3/volumes/kubernetes.io~projected/kube-api-access-7l6qp:{mountpoint:/var/lib/kubelet/pods/55f98f6e-7d08-4d15-9df2-24d5c205a5d3/volumes/kubernetes.io~projected/kube-api-access-7l6qp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~projected/kube-api-access-lxx2t:{mountpoint:/var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~projected/kube-api-access-lxx2t major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f939783-c4a2-4931-9ab9-07f56e14a7ed/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6f939783-c4a2-4931-9ab9-07f56e14a7ed/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d3b166c-1f4f-4645-bc93-d76093c412e1/volumes/kubernetes.io~projected/kube-api-access-glv87:{mountpoint:/var/lib/kubelet/pods/7d3b166c-1f4f-4645-bc93-d76093c412e1/volumes/kubernetes.io~projected/kube-api-access-glv87 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9867d6a5-3527-4058-80a0-1d8dd1e1590a/volumes/kubernetes.io~projected/kube-api-access-j9qtn:{mountpoint:/var/lib/kubelet/pods/9867d6a5-3527-4058-80a0-1d8dd1e1590a/volumes/kubernetes.io~projected/kube-api-access-j9qtn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8/volumes/kubernetes.io~projected/kube-api-access-bfpf4:{mountpoint:/var/lib/kubelet/pods/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8/volumes/kubernetes.io~projected/kube-api-access-bfpf4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a793afe1-5ad9-4746-8a70-631f91206420/volumes/kubernetes.io~projected/kube-api-access-k6f7h:{mountpoint:/var/lib/kubelet/pods/a793afe1-5ad9-4746-8a70-631f91206420/volumes/kubernetes.io~projected/kube-api-access-k6f7h major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a793afe1-5ad9-4746-8a70-631f91206420/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/a793afe1-5ad9-4746-8a70-631f91206420/volumes/kubernetes.io~secret/signing-key major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9ac25cf-7824-4fca-8c91-d3c7293e3834/volumes/kubernetes.io~projected/kube-api-access-lc9vq:{mountpoint:/var/lib/kubelet/pods/c9ac25cf-7824-4fca-8c91-d3c7293e3834/volumes/kubernetes.io~projected/kube-api-access-lc9vq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd70ad9e-6914-4bba-8e93-de02ed571a4f/volumes/kubernetes.io~projected/kube-api-access-c6jd7:{mountpoint:/var/lib/kubelet/pods/cd70ad9e-6914-4bba-8e93-de02ed571a4f/volumes/kubernetes.io~projected/kube-api-access-c6jd7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df1447be-4ab8-40c9-8606-de70d0920e65/volumes/kubernetes.io~projected/kube-api-access-h5vdf:{mountpoint:/var/lib/kubelet/pods/df1447be-4ab8-40c9-8606-de70d0920e65/volumes/kubernetes.io~projected/kube-api-access-h5vdf major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efe751b0-e2fe-4576-83de-e9dcb258291b/volumes/kubernetes.io~projected/kube-api-access-w6jfb:{mountpoint:/var/lib/kubelet/pods/efe751b0-e2fe-4576-83de-e9dcb258291b/volumes/kubernetes.io~projected/kube-api-access-w6jfb major:0 minor:53 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4eaf06482d8f03bf11451230c0a5b31ead8053abe587120c7639f97f6ecef4e4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6d685b4ab94b55180f4884f8fdeb557b9aaecf3c0c4085eb7766c9bb9aac687c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8485aacfef7e6b4979415e9e61904b09b0332f8214b476ca459cd07ea0c8ad11/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1464aac535fbe79bca11ac113d8f270943b28b561240874372bb978386ec865e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/e17004633d13082a6c09b438d5afb53680d90c8c5a67198e3c41d769d2c9c195/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/977892c60f2cf55a7045c5d0dcb6ef733d320a26a42363271171399ab92db98d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f6f57a5d21b7a31df03e77d17df1b9df4d68424e374f6a070daa629480f789a6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ba29f90d5385b7aecf5bb22e32b16b250a5389bb9ebc5405a2083531dbdeae9a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2cd8c31834dc8c316164c7dea4a1e6acaeab756c09a7674b40ca6986e2dddbe0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ae470eb66abce2ce6f805eb9167c503ffdb2e2e7564da519122286a7c707b3c9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/98ffdd76c157b00af499651be5da46c7032b9cbe0f3ca1a0957b2b23526b283f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0977790692f7d7a47225533e1666a335818deba06555cd55f4d3b61a59a6e034/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/52dace790b1698c961820cb5b6aa25e79cf8ccf956a191ab3ba183b17764ff56/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c443d84c142f84272c3bd090090d27a87f4b7f73db9aa418e77c1ba45ea7d7d0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/db60a216d922e9e44c378ab3fe85a56a0f7432c0be39fcbd1fc70be9b5f7d0a5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a3d843fb94edf9cd3290d07567a47a6d2171a380c579ee577351cb16902a7b67/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d5266ea0c482151b91cf1cea3ba92f45fe0ab0a78e5df67758b1e604f7a65e42/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5f83b0f570ad8f528b72c1d3c22bfc2b5d9e080bd2e8300ae42ed83ea3c0d657/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/fe0fccb03fcb555af8498be4dbdee293596dd9548af442e206af4252c393d9ba/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6f5895e92e7f3e9427604d9805e6c8b648c910db04b17126a451ab92e1de2b61/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ee8ac44d6425c39b1e4a649f208b3d0024f703f0fc42e2807036f313d28a771d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/074b4764b02eb7e6c67654eb71bc5e7c3460a245e73725ad6ad0ce0bd10c85ae/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/002f44a085f4ef4c21d8f81c42a85386c48ae43fe977bf9b7a0e5816caaf955b/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/147ab3f1ddeff6216cb27076ec3299f57fdf6e8cda0baaf59e897efb5e171b86/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/77d75393c3f2c0a7a717f748304080355bd9b3b04cc4fd061eab4b5fbce8f9be/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/c0f1b022c8a3f3e0db2382ef00bc7a019b5f6d476b6fb66ba4e5db15de86ff5f/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/4408e45796d35d60ad05d6e917f869769e82e0deb783f072de980b7a763bbde5/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/d1db46d640df84a32550e8985b03ef021586bd71a8645e3fb509926c4c29f701/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/3bc106af5e1e336b3e82618f0a269db6ad187dd4c1ef8a42a359be3bdd7e4454/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/d55aa51a0f98114e7764af48a6298d9047edb29d4e106881ad3ba1d730a35e38/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b14e413ee5e81f69daa8257650f93dc6f7652e42ccff6114efd3e3b1550881c8/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/fa983915729fef39bc37f2b0115de4cfe525676def08940897c07e3ba236b781/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/588b6e84d6f8c9086cffa98353b2e299a906b6d67939abfa6f1c5d391f2ed5c3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/90a47142c902f3594aede0128368249124a6ed9c04c990cc24460de10a97116b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/af8d3e6b443f9a7a5f395cd1f2c5f29bd7756aa771f01f3cc95169aa6e1ed07e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/99f216001a912bfdbcb6cea52dd95d8d98ac68ecf60082222ddd93b08c96f752/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c73a441d85c836ce6f6333229beef64df67ad62cd677630fd1ae5c02c84b0e03/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/af30c5180d506a1d3220109b8c73cf84861960a2b8c20b16d9ae31a2fa73032b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e262040956fd7c5e25225f7cea549d277684d735f45a3f232907150e95c97ed2/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c672a83ec9febe305ee91c7bbd0b25354de009fb3db8659ec9aa71427dea8afe/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.252271    4476 manager.go:217] Machine: {Timestamp:2026-11-02 17:22:49.250967561 +0000 UTC m=+0.091573486 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094956 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f9a2224369cb039ddfc06ed3c908 SystemUUID:ec24f9a2-2243-69cb-039d-dfc06ed3c908 BootID:7136298e-fa6e-46f2-b810-65039c50199f Filesystems:[{Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd79e8c8548809f704ba833a4acc10e71e6d07db4edbaaf83975e60c588316d6/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c42fdecabbf9d9920917c4021591e09b6aebd2b3a10e5e1e2fca0ccff6ac1d77/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f/volumes/kubernetes.io~projected/kube-api-access-mcsqh DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/00939ab6-259b-42cd-be51-95182c9ba479/volumes/kubernetes.io~projected/kube-api-access-hppnh DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f939783-c4a2-4931-9ab9-07f56e14a7ed/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/295a2a66d403f44a60c79066622253599a9b86a90030ccc845e726d1a926805f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1e6b3bff-315b-4fd5-b428-a2da329b0cea/volumes/kubernetes.io~projected/kube-api-access-67hqt DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a793afe1-5ad9-4746-8a70-631f91206420/volumes/kubernetes.io~projected/kube-api-access-k6f7h DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ed5f8dc4d5e38bb97a2bb1f514b2cdbdd47f8e891bee00461b0ab0dfd83dbae/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/087aba7b9163c11edeca32526893266b81702b7e9c5aee02ff57cfc38d7acb1b/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25000ba3-538e-482f-bec4-e882e1053b1f/volumes/kubernetes.io~projected/kube-api-access-r455q DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9ac25cf-7824-4fca-8c91-d3c7293e3834/volumes/kubernetes.io~projected/kube-api-access-lc9vq DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/235cadf2ceee0ab284d643cf6784bd56d4db11c6c8ebc0eb32d656c44fae9c9e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d3b166c-1f4f-4645-bc93-d76093c412e1/volumes/kubernetes.io~projected/kube-api-access-glv87 DeviceMajor:0 DeviceMinor:228 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cd70ad9e-6914-4bba-8e93-de02ed571a4f/volumes/kubernetes.io~projected/kube-api-access-c6jd7 DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/781a899c2760897325a94f5d1a8cae90111d5ec091724373f0664e5f66c6db25/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a793afe1-5ad9-4746-8a70-631f91206420/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:239 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volumes/kubernetes.io~projected/kube-api-access-9f548 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/55f98f6e-7d08-4d15-9df2-24d5c205a5d3/volumes/kubernetes.io~projected/kube-api-access-7l6qp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efe751b0-e2fe-4576-83de-e9dcb258291b/volumes/kubernetes.io~projected/kube-api-access-w6jfb DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6145fee0e84b6ab93dcbd43074f6cb91be1f3562b55a62075d24dfdd57f4ca66/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bb4434011ced7d82474632cee46681e989e4d8b7e9e6744ff4ab5766b4c50a9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/29109340-d347-45c7-a6dc-891cc6f752a9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1e6b3bff-315b-4fd5-b428-a2da329b0cea/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee60d8a9b0793f08d6e5e5d4bdfdcc1ce5eca92c9c1e4eacbd9b009618cfaf18/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df1447be-4ab8-40c9-8606-de70d0920e65/volumes/kubernetes.io~projected/kube-api-access-h5vdf DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3f80bf07-9f23-46af-b53f-35bbd3157b6d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/6a5b0c75-fe80-4472-b423-97785579f253/volumes/kubernetes.io~projected/kube-api-access-lxx2t DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdf738a3026bb44a50f215e6c3d91bec3147aa3404c588a115fd44d046b379c3/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4364361049f49f5b978aa66d5c4f1ef4f908b55f7f7f9c2bcbad7a01b969c964/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8/volumes/kubernetes.io~projected/kube-api-access-bfpf4 DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19284cc9-6243-40af-862b-dc011abb8b7c/volumes/kubernetes.io~projected/kube-api-access-jtfs8 DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35e3226cbbe68ab4d6dfc14df412ec2b1520ccf286582a4fab39e80bcbfe8a80/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9867d6a5-3527-4058-80a0-1d8dd1e1590a/volumes/kubernetes.io~projected/kube-api-access-j9qtn DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:33504445c6b2236 MacAddress:6a:03:2f:b4:76:ea Speed:10000 Mtu:8901} {Name:4364361049f49f5 MacAddress:6a:89:2b:d7:e2:cd Speed:10000 Mtu:8901} {Name:781a899c2760897 MacAddress:0e:1e:6a:6d:ef:2d Speed:10000 Mtu:8901} {Name:9ed5f8dc4d5e38b MacAddress:c2:1b:40:51:99:d2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3c:8c:cd:a6:a5 Speed:0 Mtu:9001} {Name:br-int MacAddress:92:29:ed:70:dd:6c Speed:0 Mtu:8901} {Name:ee60d8a9b0793f0 MacAddress:ae:60:98:76:ad:ec Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3c:8c:cd:a6:a5 Speed:0 Mtu:9001} {Name:fd79e8c8548809f MacAddress:c6:09:e5:4a:bb:23 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f6:25:80:db:b9:f7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:80:58:c5:e3:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253124    4476 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253186    4476 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253450    4476 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253467    4476 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-192.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253592    4476 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253600    4476 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253611    4476 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253649    4476 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253749    4476 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.253902    4476 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254003    4476 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254036    4476 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254050    4476 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254064    4476 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254073    4476 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254082    4476 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254649    4476 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.254659    4476 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:49.255362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.255108    4476 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:22:49.336129 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336112    4476 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:22:49.336192 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336129    4476 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:22:49.336263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336252    4476 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 17:22:49.336479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336468    4476 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:22:49.336510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336498    4476 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:22:49.336650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336621    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:22:49.336704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336660    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:22:49.336704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336669    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:22:49.336704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336675    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:22:49.336704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336684    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:22:49.336704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336692    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336707    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336713    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336720    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336732    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336746    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336756    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336783    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:22:49.336890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.336789    4476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:22:49.340415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.340371    4476 server.go:1263] "Started kubelet"
Nov 02 17:22:49.340679 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.340467    4476 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:22:49.341153 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.341050    4476 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:22:49.341254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.341181    4476 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:22:49.341566 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.341546    4476 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:22:49.341727 ip-10-0-136-192 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:22:49.342119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.341942    4476 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.344395    4476 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.344672    4476 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.344685    4476 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.344850    4476 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.344993    4476 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.345187    4476 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-192.us-west-1.compute.internal\" not found"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.345353    4476 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:11 +0000 UTC" deadline="2028-05-01 16:30:31.903427993 +0000 UTC"
Nov 02 17:22:49.346957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.345391    4476 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13103h7m42.558040681s"
Nov 02 17:22:49.348172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.347927    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29109340-d347-45c7-a6dc-891cc6f752a9" volumeName="kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 17:22:49.348353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348322    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f939783-c4a2-4931-9ab9-07f56e14a7ed" volumeName="kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348355    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" volumeName="kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348371    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19284cc9-6243-40af-862b-dc011abb8b7c" volumeName="kubernetes.io/configmap/19284cc9-6243-40af-862b-dc011abb8b7c-config-volume" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348384    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f98f6e-7d08-4d15-9df2-24d5c205a5d3" volumeName="kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-daemon-config" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348397    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df1447be-4ab8-40c9-8606-de70d0920e65" volumeName="kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348410    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df1447be-4ab8-40c9-8606-de70d0920e65" volumeName="kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348421    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e6b3bff-315b-4fd5-b428-a2da329b0cea" volumeName="kubernetes.io/projected/1e6b3bff-315b-4fd5-b428-a2da329b0cea-kube-api-access-67hqt" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348437    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f80bf07-9f23-46af-b53f-35bbd3157b6d" volumeName="kubernetes.io/secret/3f80bf07-9f23-46af-b53f-35bbd3157b6d-agent-certs" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348449    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a793afe1-5ad9-4746-8a70-631f91206420" volumeName="kubernetes.io/projected/a793afe1-5ad9-4746-8a70-631f91206420-kube-api-access-k6f7h" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348461    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" volumeName="kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348473    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29109340-d347-45c7-a6dc-891cc6f752a9" volumeName="kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-env-overrides" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348499    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5b0c75-fe80-4472-b423-97785579f253" volumeName="kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-tmp" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348511    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb3b92e-c809-4e8d-a886-9d3ef6b35da8" volumeName="kubernetes.io/empty-dir/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-data-volume" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348527    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a793afe1-5ad9-4746-8a70-631f91206420" volumeName="kubernetes.io/configmap/a793afe1-5ad9-4746-8a70-631f91206420-signing-cabundle" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348539    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efe751b0-e2fe-4576-83de-e9dcb258291b" volumeName="kubernetes.io/projected/efe751b0-e2fe-4576-83de-e9dcb258291b-kube-api-access-w6jfb" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348551    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29109340-d347-45c7-a6dc-891cc6f752a9" volumeName="kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-config" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348563    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5b0c75-fe80-4472-b423-97785579f253" volumeName="kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-etc-tuned" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348575    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9867d6a5-3527-4058-80a0-1d8dd1e1590a" volumeName="kubernetes.io/configmap/9867d6a5-3527-4058-80a0-1d8dd1e1590a-serviceca" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348587    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9ac25cf-7824-4fca-8c91-d3c7293e3834" volumeName="kubernetes.io/empty-dir/c9ac25cf-7824-4fca-8c91-d3c7293e3834-tmp-dir" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348599    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd70ad9e-6914-4bba-8e93-de02ed571a4f" volumeName="kubernetes.io/projected/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kube-api-access-c6jd7" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348611    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df1447be-4ab8-40c9-8606-de70d0920e65" volumeName="kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348645    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19284cc9-6243-40af-862b-dc011abb8b7c" volumeName="kubernetes.io/projected/19284cc9-6243-40af-862b-dc011abb8b7c-kube-api-access-jtfs8" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348661    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e6b3bff-315b-4fd5-b428-a2da329b0cea" volumeName="kubernetes.io/secret/1e6b3bff-315b-4fd5-b428-a2da329b0cea-cert" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348673    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29109340-d347-45c7-a6dc-891cc6f752a9" volumeName="kubernetes.io/projected/29109340-d347-45c7-a6dc-891cc6f752a9-kube-api-access-9f548" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348685    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f80bf07-9f23-46af-b53f-35bbd3157b6d" volumeName="kubernetes.io/configmap/3f80bf07-9f23-46af-b53f-35bbd3157b6d-konnectivity-ca" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348695    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f98f6e-7d08-4d15-9df2-24d5c205a5d3" volumeName="kubernetes.io/projected/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-kube-api-access-7l6qp" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348707    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9eb3b92e-c809-4e8d-a886-9d3ef6b35da8" volumeName="kubernetes.io/projected/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-api-access-bfpf4" seLinuxMountContext=""
Nov 02 17:22:49.348704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348724    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a793afe1-5ad9-4746-8a70-631f91206420" volumeName="kubernetes.io/secret/a793afe1-5ad9-4746-8a70-631f91206420-signing-key" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348762    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df1447be-4ab8-40c9-8606-de70d0920e65" volumeName="kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348775    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00939ab6-259b-42cd-be51-95182c9ba479" volumeName="kubernetes.io/projected/00939ab6-259b-42cd-be51-95182c9ba479-kube-api-access-hppnh" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348790    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25000ba3-538e-482f-bec4-e882e1053b1f" volumeName="kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348802    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9867d6a5-3527-4058-80a0-1d8dd1e1590a" volumeName="kubernetes.io/projected/9867d6a5-3527-4058-80a0-1d8dd1e1590a-kube-api-access-j9qtn" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348815    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efe751b0-e2fe-4576-83de-e9dcb258291b" volumeName="kubernetes.io/configmap/efe751b0-e2fe-4576-83de-e9dcb258291b-iptables-alerter-script" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348827    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d3b166c-1f4f-4645-bc93-d76093c412e1" volumeName="kubernetes.io/projected/7d3b166c-1f4f-4645-bc93-d76093c412e1-kube-api-access-glv87" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348841    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" volumeName="kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348853    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" volumeName="kubernetes.io/projected/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-kube-api-access-mcsqh" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348865    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29109340-d347-45c7-a6dc-891cc6f752a9" volumeName="kubernetes.io/secret/29109340-d347-45c7-a6dc-891cc6f752a9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348873    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f98f6e-7d08-4d15-9df2-24d5c205a5d3" volumeName="kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348882    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a5b0c75-fe80-4472-b423-97785579f253" volumeName="kubernetes.io/projected/6a5b0c75-fe80-4472-b423-97785579f253-kube-api-access-lxx2t" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348894    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9ac25cf-7824-4fca-8c91-d3c7293e3834" volumeName="kubernetes.io/projected/c9ac25cf-7824-4fca-8c91-d3c7293e3834-kube-api-access-lc9vq" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348904    4476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19284cc9-6243-40af-862b-dc011abb8b7c" volumeName="kubernetes.io/empty-dir/19284cc9-6243-40af-862b-dc011abb8b7c-tmp-dir" seLinuxMountContext=""
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348914    4476 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.348922    4476 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.349818    4476 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.349833    4476 factory.go:55] Registering systemd factory
Nov 02 17:22:49.349886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.349843    4476 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:22:49.350517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.350358    4476 factory.go:153] Registering CRI-O factory
Nov 02 17:22:49.350517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.350371    4476 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:22:49.350517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.350389    4476 factory.go:103] Registering Raw factory
Nov 02 17:22:49.350517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.350399    4476 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:22:49.350745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.350733    4476 manager.go:319] Starting recovery of all containers
Nov 02 17:22:49.357156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.357116    4476 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:22:49.375127 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.375096    4476 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:22:49.375127 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.375121    4476 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:22:49.375287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.375137    4476 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:22:49.375287 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.375175    4476 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 17:22:49.396869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.396834    4476 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="103cbcc3b0d755296179cc55e4235112f2adbbf623d35c9f8acb9a74031e5722" exitCode=0
Nov 02 17:22:49.396869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.396866    4476 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="92c7add7d1ee37683687e8187601e6f6814228a3eff53cded7a17afff27d6f6f" exitCode=0
Nov 02 17:22:49.396869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.396879    4476 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="48a65da30606e58bd584debbf7c277ba20a683fa45fbf9cb1f9028844c983598" exitCode=0
Nov 02 17:22:49.397122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.396888    4476 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="f6a671bf3f6790c04061372789f370d9d44ad578f86608a1d26c31b102ff9b9a" exitCode=0
Nov 02 17:22:49.397122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.396897    4476 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="3fed9542f4f7eab84b32b14e0e5e7156a5083523fa6353a0b826ed263c8cf35c" exitCode=0
Nov 02 17:22:49.397122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.396906    4476 generic.go:356] "Generic (PLEG): container finished" podID="0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f" containerID="2c482d5d22369a9dc63a412c8ba11dacd07eddd9945ca3d0efc23551d6e6e462" exitCode=0
Nov 02 17:22:49.399417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.399389    4476 generic.go:356] "Generic (PLEG): container finished" podID="ab93effd0670cefe2422e74f59f96c7d" containerID="7312ddb41b161c089c167135c7936b922429a552bce0af2615e978ca1360ecf1" exitCode=0
Nov 02 17:22:49.418095 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.418074    4476 manager.go:324] Recovery completed
Nov 02 17:22:49.431294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.431271    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.432501 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.432481    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.432613 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.432517    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.432613 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.432531    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.435917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.435898    4476 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:22:49.435917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.435914    4476 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:22:49.436080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.435935    4476 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:22:49.436080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.435903    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.436172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436093    4476 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 17:22:49.436172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436127    4476 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 17:22:49.436172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436153    4476 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 17:22:49.436172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436162    4476 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 17:22:49.436172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436170    4476 policy_none.go:50] "Start"
Nov 02 17:22:49.436394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436182    4476 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:22:49.436394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436195    4476 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:22:49.436394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436375    4476 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 17:22:49.436394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436386    4476 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 17:22:49.436394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436395    4476 policy_none.go:44] "Start"
Nov 02 17:22:49.436831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436817    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.436901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436844    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.436901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.436854    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.440529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.440504    4476 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:22:49.444191 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444175    4476 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:22:49.444280 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.444202    4476 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:22:49.444280 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444213    4476 server.go:87] "Starting device plugin registration server"
Nov 02 17:22:49.444508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444494    4476 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:22:49.444567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444508    4476 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:22:49.444658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444609    4476 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:22:49.444759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444723    4476 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:22:49.444820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444769    4476 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:22:49.444820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444775    4476 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:22:49.445553 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.445529    4476 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-192.us-west-1.compute.internal\" not found"
Nov 02 17:22:49.445671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.444837    4476 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:22:49.44475797Z","UUID":"e75e0c45-48e5-4fb7-a8d9-4746dcaf5b28","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:22:49.446059 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.446040    4476 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:22:49.446144 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.446084    4476 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-192.us-west-1.compute.internal\" not found"
Nov 02 17:22:49.446978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.446955    4476 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:22:49.447056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.447001    4476 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:22:49.462753 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.462728    4476 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:49.462873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.462797    4476 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 17:22:49.463214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.463201    4476 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:49.463764 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.463751    4476 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:49.475729 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.475691    4476 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"]
Nov 02 17:22:49.476292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476261    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"2661666f349d448cd741b0565fc226b6","Type":"ContainerStarted","Data":"e6b0eab0da957b8535801ed79eb9c38bf5d432fffad1ea99b81c680818310926"}
Nov 02 17:22:49.476330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476298    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"2661666f349d448cd741b0565fc226b6","Type":"ContainerStarted","Data":"258c2cdb24732f9fd662752dd7cb094ed6515f9f499592e5c519ef7bf636ca03"}
Nov 02 17:22:49.476384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476335    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.476384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476348    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"ab93effd0670cefe2422e74f59f96c7d","Type":"ContainerStarted","Data":"9ead4309da96b5fa395b6d2500970a2c0cd325c34ce533dbd3560bf0c47eaf59"}
Nov 02 17:22:49.476384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476361    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"ab93effd0670cefe2422e74f59f96c7d","Type":"ContainerDied","Data":"7312ddb41b161c089c167135c7936b922429a552bce0af2615e978ca1360ecf1"}
Nov 02 17:22:49.476514 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476389    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal" event={"ID":"ab93effd0670cefe2422e74f59f96c7d","Type":"ContainerStarted","Data":"c585d85ae2d641a3cde752586ad25b4f6c4aa6874e510caf2d407abc60bcf32d"}
Nov 02 17:22:49.476600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.476587    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.514421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.514394    4476 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:22:49.545907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.545878    4476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:22:49.546045 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.545920    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.546947 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.546933    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:22:49.547007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.546963    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:22:49.547007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.546975    4476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-192.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:22:49.547086 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.547018    4476 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.560925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.560902    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:49.561006 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.560952    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:49.561006 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.560972    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.561006 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.561002    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.563137 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.563117    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.563194 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.563145    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.563194 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.563162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2661666f349d448cd741b0565fc226b6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"2661666f349d448cd741b0565fc226b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.605052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.604986    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:49.605052 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.605031    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.605052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.605045    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.656204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.656181    4476 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.656265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.656248    4476 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.663350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.663323    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.663457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.663352    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.663457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.663373    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2661666f349d448cd741b0565fc226b6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"2661666f349d448cd741b0565fc226b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.663457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.663423    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2661666f349d448cd741b0565fc226b6-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"2661666f349d448cd741b0565fc226b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.663457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.663443    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.663613 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.663423    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab93effd0670cefe2422e74f59f96c7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" (UID: \"ab93effd0670cefe2422e74f59f96c7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:49.666525 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:49.666511    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:49.666575 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:49.666554    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:50.255293 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.255253    4476 apiserver.go:52] "Watching apiserver"
Nov 02 17:22:50.358814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.358787    4476 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:22:50.359319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.359297    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-trcmz","openshift-ovn-kubernetes/ovnkube-node-mhl5c","kube-system/kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj","openshift-ingress-canary/ingress-canary-rxx72","openshift-insights/insights-runtime-extractor-pmnsd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal","openshift-image-registry/node-ca-nqt8d","openshift-multus/multus-4df8n","openshift-multus/multus-additional-cni-plugins-9spt5","openshift-network-operator/iptables-alerter-tw9mq","openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv","openshift-multus/network-metrics-daemon-2b8zt","openshift-network-diagnostics/network-check-target-slw2l","openshift-service-ca/service-ca-5fffc97bdf-tf45l","openshift-insights/periodic-gathering-kxkns-mjrds","openshift-dns/dns-default-n8b9d","kube-system/konnectivity-agent-dz5j7","openshift-cluster-node-tuning-operator/tuned-pmr9j","openshift-dns/node-resolver-hs745"]
Nov 02 17:22:50.359752 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.359734    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:50.363777 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.363757    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.364588 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.364569    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.414179 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.414135    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:22:50.414511 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.414487    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:22:50.415247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415225    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:22:50.415485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415250    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\""
Nov 02 17:22:50.415592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415440    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-546w9\""
Nov 02 17:22:50.415719 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415499    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkkxw\""
Nov 02 17:22:50.415791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415722    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-867zk\""
Nov 02 17:22:50.415791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415781    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:22:50.415974 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.415956    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wf5x\""
Nov 02 17:22:50.416337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.416314    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztdmt\""
Nov 02 17:22:50.416447 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.416430    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:22:50.416621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.416609    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:22:50.416685 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.416663    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2rxq\""
Nov 02 17:22:50.417319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417296    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:22:50.417393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417358    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.417441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417400    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.417507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417492    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.417703 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417687    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jxqfl\""
Nov 02 17:22:50.417767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417727    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\""
Nov 02 17:22:50.417767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417733    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4464\""
Nov 02 17:22:50.417857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417781    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:22:50.417913 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417895    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:22:50.417958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417896    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:22:50.417958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417935    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.417958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.417946    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:22:50.418227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418075    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.418227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418085    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:22:50.418227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418139    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:22:50.418227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418190    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:22:50.418392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418228    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.418392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418273    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.418392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418382    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:22:50.418565 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418546    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:22:50.418641 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418590    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:22:50.418776 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418548    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.418834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418815    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.418834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418823    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.418834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.418547    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b28rw\""
Nov 02 17:22:50.419374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419354    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.419374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419370    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.419475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419384    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.419475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419421    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kdpj\""
Nov 02 17:22:50.419475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419439    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.419595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419473    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.419595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.419519    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\""
Nov 02 17:22:50.421099 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.421079    4476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:50.446393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.446372    4476 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:22:50.463506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.463484    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svxv2\""
Nov 02 17:22:50.463578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.463541    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.463578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.463551    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:22:50.463578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.463570    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.467799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467781    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:50.467854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467808    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ac25cf-7824-4fca-8c91-d3c7293e3834-tmp-dir\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.467854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467826    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19284cc9-6243-40af-862b-dc011abb8b7c-tmp-dir\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.467854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467848    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-script-lib\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467872    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6qp\" (UniqueName: \"kubernetes.io/projected/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-kube-api-access-7l6qp\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.468011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467901    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9vq\" (UniqueName: \"kubernetes.io/projected/c9ac25cf-7824-4fca-8c91-d3c7293e3834-kube-api-access-lc9vq\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.468011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467928    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppnh\" (UniqueName: \"kubernetes.io/projected/00939ab6-259b-42cd-be51-95182c9ba479-kube-api-access-hppnh\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:50.468011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467953    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysconfig\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.468011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467959    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ac25cf-7824-4fca-8c91-d3c7293e3834-tmp-dir\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.468011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467970    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19284cc9-6243-40af-862b-dc011abb8b7c-tmp-dir\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.467978    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-etc-tuned\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468072    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468098    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-var-lib-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468140    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-etc-tuned\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468145    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468203    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f548\" (UniqueName: \"kubernetes.io/projected/29109340-d347-45c7-a6dc-891cc6f752a9-kube-api-access-9f548\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468246    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.468286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468271    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-script-lib\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468275    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f80bf07-9f23-46af-b53f-35bbd3157b6d-konnectivity-ca\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468319    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-modprobe-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468340    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-host\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468395    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-systemd-units\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468432    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-systemd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468457    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468503    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29109340-d347-45c7-a6dc-891cc6f752a9-ovn-node-metrics-cert\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468537    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-sys-fs\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468559    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-run\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468587    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19284cc9-6243-40af-862b-dc011abb8b7c-config-volume\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468609    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f80bf07-9f23-46af-b53f-35bbd3157b6d-konnectivity-ca\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:50.468710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468653    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-ovn\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468754    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29109340-d347-45c7-a6dc-891cc6f752a9-ovn-node-metrics-cert\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468764    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-kubelet-config\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468798    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-netns\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468811    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19284cc9-6243-40af-862b-dc011abb8b7c-config-volume\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468824    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-conf\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468849    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-data-volume\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468881    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-bin\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468919    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-registration-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468924    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-data-volume\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468940    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jd7\" (UniqueName: \"kubernetes.io/projected/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kube-api-access-c6jd7\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.469034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.468960    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hqt\" (UniqueName: \"kubernetes.io/projected/1e6b3bff-315b-4fd5-b428-a2da329b0cea-kube-api-access-67hqt\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469052    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsqh\" (UniqueName: \"kubernetes.io/projected/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-kube-api-access-mcsqh\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469081    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a793afe1-5ad9-4746-8a70-631f91206420-signing-key\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469096    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfs8\" (UniqueName: \"kubernetes.io/projected/19284cc9-6243-40af-862b-dc011abb8b7c-kube-api-access-jtfs8\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469122    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efe751b0-e2fe-4576-83de-e9dcb258291b-iptables-alerter-script\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469232    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469257    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469309    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9867d6a5-3527-4058-80a0-1d8dd1e1590a-serviceca\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469330    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxx2t\" (UniqueName: \"kubernetes.io/projected/6a5b0c75-fe80-4472-b423-97785579f253-kube-api-access-lxx2t\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469348    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469362    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-log-socket\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469386    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-netd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469470    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efe751b0-e2fe-4576-83de-e9dcb258291b-iptables-alerter-script\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469495    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-config\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469533    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469535    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9867d6a5-3527-4058-80a0-1d8dd1e1590a-serviceca\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469560    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cnibin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469609    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e6b3bff-315b-4fd5-b428-a2da329b0cea-cert\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469657    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f80bf07-9f23-46af-b53f-35bbd3157b6d-agent-certs\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469717    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-ovnkube-config\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469766    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469805    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-kubelet\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469833    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-etc-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469847    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e6b3bff-315b-4fd5-b428-a2da329b0cea-cert\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:50.469861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469858    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469892    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-kubelet\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469914    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469916    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f80bf07-9f23-46af-b53f-35bbd3157b6d-agent-certs\") pod \"konnectivity-agent-dz5j7\" (UID: \"3f80bf07-9f23-46af-b53f-35bbd3157b6d\") " pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469931    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv87\" (UniqueName: \"kubernetes.io/projected/7d3b166c-1f4f-4645-bc93-d76093c412e1-kube-api-access-glv87\") pod \"migrator-66df5d496b-jjlxv\" (UID: \"7d3b166c-1f4f-4645-bc93-d76093c412e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469949    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-netns\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469966    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.469984    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-node-log\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470023    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-env-overrides\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470053    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-socket-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470102    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-dbus\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470129    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-socket-dir-parent\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470150    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29109340-d347-45c7-a6dc-891cc6f752a9-env-overrides\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470159    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-multus-certs\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470184    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470218    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.470244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470239    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-k8s-cni-cncf-io\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470257    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-conf-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470262    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470272    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9867d6a5-3527-4058-80a0-1d8dd1e1590a-host\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470316    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qtn\" (UniqueName: \"kubernetes.io/projected/9867d6a5-3527-4058-80a0-1d8dd1e1590a-kube-api-access-j9qtn\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470344    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-crio-socket\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470360    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470394    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efe751b0-e2fe-4576-83de-e9dcb258291b-host-slash\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470416    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-etc-kubernetes\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470462    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9ac25cf-7824-4fca-8c91-d3c7293e3834-hosts-file\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470506    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470550    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cni-binary-copy\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.470747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470658    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-bin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470753    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f939783-c4a2-4931-9ab9-07f56e14a7ed-original-pull-secret\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470766    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-hostroot\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470812    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-kubernetes\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470820    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cni-binary-copy\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470834    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-slash\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470863    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-daemon-config\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470888    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-systemd\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470903    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-var-lib-kubelet\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470919    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpf4\" (UniqueName: \"kubernetes.io/projected/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-api-access-bfpf4\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.470950    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jfb\" (UniqueName: \"kubernetes.io/projected/efe751b0-e2fe-4576-83de-e9dcb258291b-kube-api-access-w6jfb\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471030    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-device-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.471073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471074    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-system-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471118    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-multus\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471137    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-system-cni-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471142    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-daemon-config\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471185    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471213    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471246    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-sys\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471270    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-lib-modules\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471299    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471328    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471349    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-binary-copy\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471382    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471420    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cnibin\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471443    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.471723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471462    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-tmp\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471521    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a5b0c75-fe80-4472-b423-97785579f253-tmp\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.471723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471528    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a793afe1-5ad9-4746-8a70-631f91206420-signing-cabundle\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:50.471723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.471563    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6f7h\" (UniqueName: \"kubernetes.io/projected/a793afe1-5ad9-4746-8a70-631f91206420-kube-api-access-k6f7h\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:50.482483 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.482464    4476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:22:50.482589 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.482514    4476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.us-west-1.compute.internal"
Nov 02 17:22:50.513014 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.512934    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 17:22:50.513014 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.512941    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:22:50.513014 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.512955    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wfdxg\""
Nov 02 17:22:50.513207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.513034    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xdlgk\""
Nov 02 17:22:50.513207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.513087    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2zcw\""
Nov 02 17:22:50.513207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.513156    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gfr4c\""
Nov 02 17:22:50.514306 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514267    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.514306 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514292    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:22:50.514306 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514310    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 17:22:50.514513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514319    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.514513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514329    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.514513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514352    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:22:50.514513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514361    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.514513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514362    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.514513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514383    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:22:50.514752 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.514672    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:22:50.515353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.515338    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:22:50.519181 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.519166    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.519328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.519314    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a793afe1-5ad9-4746-8a70-631f91206420-signing-key\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:50.519539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.519524    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.522478 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.522460    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a793afe1-5ad9-4746-8a70-631f91206420-signing-cabundle\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:50.524324 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.524302    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f548\" (UniqueName: \"kubernetes.io/projected/29109340-d347-45c7-a6dc-891cc6f752a9-kube-api-access-9f548\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.524445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.524429    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppnh\" (UniqueName: \"kubernetes.io/projected/00939ab6-259b-42cd-be51-95182c9ba479-kube-api-access-hppnh\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:50.524693 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.524677    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r455q\" (UniqueName: \"kubernetes.io/projected/25000ba3-538e-482f-bec4-e882e1053b1f-kube-api-access-r455q\") pod \"network-check-target-slw2l\" (UID: \"25000ba3-538e-482f-bec4-e882e1053b1f\") " pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:50.524854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.524833    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9vq\" (UniqueName: \"kubernetes.io/projected/c9ac25cf-7824-4fca-8c91-d3c7293e3834-kube-api-access-lc9vq\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.524990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.524978    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6qp\" (UniqueName: \"kubernetes.io/projected/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-kube-api-access-7l6qp\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.527370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.527354    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qtn\" (UniqueName: \"kubernetes.io/projected/9867d6a5-3527-4058-80a0-1d8dd1e1590a-kube-api-access-j9qtn\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.527409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.527393    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfs8\" (UniqueName: \"kubernetes.io/projected/19284cc9-6243-40af-862b-dc011abb8b7c-kube-api-access-jtfs8\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.527955 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.527934    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jfb\" (UniqueName: \"kubernetes.io/projected/efe751b0-e2fe-4576-83de-e9dcb258291b-kube-api-access-w6jfb\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.529437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.529421    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") pod \"periodic-gathering-kxkns-mjrds\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") " pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:22:50.529545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.529528    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6f7h\" (UniqueName: \"kubernetes.io/projected/a793afe1-5ad9-4746-8a70-631f91206420-kube-api-access-k6f7h\") pod \"service-ca-5fffc97bdf-tf45l\" (UID: \"a793afe1-5ad9-4746-8a70-631f91206420\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l"
Nov 02 17:22:50.530115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.530095    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv87\" (UniqueName: \"kubernetes.io/projected/7d3b166c-1f4f-4645-bc93-d76093c412e1-kube-api-access-glv87\") pod \"migrator-66df5d496b-jjlxv\" (UID: \"7d3b166c-1f4f-4645-bc93-d76093c412e1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv"
Nov 02 17:22:50.530150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.530114    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jd7\" (UniqueName: \"kubernetes.io/projected/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kube-api-access-c6jd7\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.530150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.530097    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpf4\" (UniqueName: \"kubernetes.io/projected/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-api-access-bfpf4\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.530226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.530168    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxx2t\" (UniqueName: \"kubernetes.io/projected/6a5b0c75-fe80-4472-b423-97785579f253-kube-api-access-lxx2t\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.531387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.531367    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsqh\" (UniqueName: \"kubernetes.io/projected/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-kube-api-access-mcsqh\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.531491 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.531473    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hqt\" (UniqueName: \"kubernetes.io/projected/1e6b3bff-315b-4fd5-b428-a2da329b0cea-kube-api-access-67hqt\") pod \"ingress-canary-rxx72\" (UID: \"1e6b3bff-315b-4fd5-b428-a2da329b0cea\") " pod="openshift-ingress-canary/ingress-canary-rxx72"
Nov 02 17:22:50.571822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571789    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571829    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571853    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-log-socket\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571881    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-netd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571899    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571904    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571929    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-log-socket\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.571934    4476 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:22:50.571979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571977    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cnibin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571995    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572000    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-kubelet\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.571941    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-netd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572025    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-kubelet\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.572045    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.072023963 +0000 UTC m=+1.912629894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : configmap "kube-rbac-proxy" not found
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572072    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-cnibin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572073    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-etc-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572108    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572109    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-etc-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572132    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-kubelet\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572157    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572170    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572181    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-netns\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572202    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-node-log\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572206    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-kubelet\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572223    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-socket-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572244    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-dbus\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572247    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-netns\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572266    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-socket-dir-parent\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572273    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-node-log\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572289    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-multus-certs\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572297    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572308    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-socket-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572318    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-dbus\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572329    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572337    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-multus-certs\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572344    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-socket-dir-parent\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.572348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572359    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-k8s-cni-cncf-io\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572381    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-conf-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572394    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9867d6a5-3527-4058-80a0-1d8dd1e1590a-host\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572411    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-crio-socket\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572426    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572429    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9867d6a5-3527-4058-80a0-1d8dd1e1590a-host\") pod \"node-ca-nqt8d\" (UID: \"9867d6a5-3527-4058-80a0-1d8dd1e1590a\") " pod="openshift-image-registry/node-ca-nqt8d"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572441    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efe751b0-e2fe-4576-83de-e9dcb258291b-host-slash\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572444    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572449    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-k8s-cni-cncf-io\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572423    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-multus-conf-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572456    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-etc-kubernetes\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572485    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efe751b0-e2fe-4576-83de-e9dcb258291b-host-slash\") pod \"iptables-alerter-tw9mq\" (UID: \"efe751b0-e2fe-4576-83de-e9dcb258291b\") " pod="openshift-network-operator/iptables-alerter-tw9mq"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572490    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-etc-kubernetes\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572495    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9ac25cf-7824-4fca-8c91-d3c7293e3834-hosts-file\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-crio-socket\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.572527    4476 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572538    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9ac25cf-7824-4fca-8c91-d3c7293e3834-hosts-file\") pod \"node-resolver-hs745\" (UID: \"c9ac25cf-7824-4fca-8c91-d3c7293e3834\") " pod="openshift-dns/node-resolver-hs745"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572553    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-bin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.572587    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.072573922 +0000 UTC m=+1.913179861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572527    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-bin\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572607    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-hostroot\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572623    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-kubernetes\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572657    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-slash\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572672    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-systemd\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572686    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-var-lib-kubelet\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-device-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572712    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-hostroot\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572731    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-system-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572756    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-device-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572758    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-multus\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572755    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-systemd\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572756    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-var-lib-kubelet\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572771    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-slash\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572730    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-kubernetes\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572773    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-system-cni-dir\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572794    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-var-lib-cni-multus\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572793    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-system-cni-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572825    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-system-cni-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572833    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-sys\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572857    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-sys\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572857    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-lib-modules\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572882    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572911    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cnibin\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572926    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572936    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-lib-modules\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572947    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysconfig\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572966    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572983    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-var-lib-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572998    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573016    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-modprobe-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.572966    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-cnibin\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573034    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysconfig\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573037    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-host\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.573026    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.573077    4476 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.573089    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.07307538 +0000 UTC m=+1.913681317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : secret "metrics-daemon-secret" not found
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573088    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-var-lib-openvswitch\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573048    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9spt5\" (UID: \"0a79d8cf-3095-4d18-9cc4-9aa8dc6ef57f\") " pod="openshift-multus/multus-additional-cni-plugins-9spt5"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573076    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-systemd-units\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573091    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-host\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:50.573108    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:51.073098215 +0000 UTC m=+1.913704136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573105    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-systemd-units\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573086    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573152    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-modprobe-d\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573171    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-systemd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573188    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573204    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-sys-fs\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573241    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-run\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573253    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-systemd\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573260    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573264    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-ovn\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573287    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-sys-fs\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573295    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-kubelet-config\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573298    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-run-ovn\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-run\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573327    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-netns\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573343    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-conf\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573342    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f939783-c4a2-4931-9ab9-07f56e14a7ed-kubelet-config\") pod \"global-pull-secret-syncer-trcmz\" (UID: \"6f939783-c4a2-4931-9ab9-07f56e14a7ed\") " pod="kube-system/global-pull-secret-syncer-trcmz"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573358    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-bin\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573377    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55f98f6e-7d08-4d15-9df2-24d5c205a5d3-host-run-netns\") pod \"multus-4df8n\" (UID: \"55f98f6e-7d08-4d15-9df2-24d5c205a5d3\") " pod="openshift-multus/multus-4df8n"
Nov 02 17:22:50.573330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573380    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-registration-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.575513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573411    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29109340-d347-45c7-a6dc-891cc6f752a9-host-cni-bin\") pod \"ovnkube-node-mhl5c\" (UID: \"29109340-d347-45c7-a6dc-891cc6f752a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:50.575513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573435    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd70ad9e-6914-4bba-8e93-de02ed571a4f-registration-dir\") pod \"aws-ebs-csi-driver-node-fxxzj\" (UID: \"cd70ad9e-6914-4bba-8e93-de02ed571a4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fxxzj"
Nov 02 17:22:50.575513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.573469    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a5b0c75-fe80-4472-b423-97785579f253-etc-sysctl-conf\") pod \"tuned-pmr9j\" (UID: \"6a5b0c75-fe80-4472-b423-97785579f253\") " pod="openshift-cluster-node-tuning-operator/tuned-pmr9j"
Nov 02 17:22:50.667237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.667217    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:22:50.878550 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:50.878468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:51.076947 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:51.076908    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:51.077154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:51.076966    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:51.077154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:51.077008    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:51.077154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:51.077046    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:51.077154 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:51.077078    4476 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:51.077154 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:51.077132    4476 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:51.077154 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:51.077141    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:51.077359 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:51.077163    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.077146025 +0000 UTC m=+2.917751937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:51.077359 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:51.077190    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.077177605 +0000 UTC m=+2.917783547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:51.077359 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:51.077205    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:52.077198328 +0000 UTC m=+2.917804241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : secret "metrics-daemon-secret" not found
Nov 02 17:22:51.077539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:51.077495    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:52.084899 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.084861    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.084919    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.084948    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:52.085033    4476 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:52.085057    4476 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:52.085067    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:52.085119    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.085098095 +0000 UTC m=+4.925704022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:52.085139    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.085129178 +0000 UTC m=+4.925735100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : secret "metrics-daemon-secret" not found
Nov 02 17:22:52.085291 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:52.085155    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:54.085147241 +0000 UTC m=+4.925753203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:52.384772 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.384698    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:52.427394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.427350    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kxkns-mjrds" event={"ID":"df1447be-4ab8-40c9-8606-de70d0920e65","Type":"ContainerStarted","Data":"aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8"}
Nov 02 17:22:52.428902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.428879    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jjlxv" event={"ID":"7d3b166c-1f4f-4645-bc93-d76093c412e1","Type":"ContainerStarted","Data":"d6f219c0c21a4afb6e31a1c60ab7b1788fc2008e22f585a788b7f62ef245f44a"}
Nov 02 17:22:52.589032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.588959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:52.771927 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.771854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:52.972450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:52.972419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:53.704917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:53.704881    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:54.102292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:54.102255    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:54.102303    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:54.102338    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:54.102406    4476 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:54.102466    4476 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:54.102483    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.102466191 +0000 UTC m=+8.943072119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:54.102512    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs podName:00939ab6-259b-42cd-be51-95182c9ba479 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.10250174 +0000 UTC m=+8.943107652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs") pod "network-metrics-daemon-2b8zt" (UID: "00939ab6-259b-42cd-be51-95182c9ba479") : secret "metrics-daemon-secret" not found
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:54.102466    4476 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:54.102580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:54.102543    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:22:58.102533987 +0000 UTC m=+8.943139902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:54.436226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:54.436191    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tf45l" event={"ID":"a793afe1-5ad9-4746-8a70-631f91206420","Type":"ContainerStarted","Data":"3ecae9d380a2e0ce058c187f1fc80b1ed9bf836871f684f701506c0f4b2240e9"}
Nov 02 17:22:54.914860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:54.914827    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:55.379848 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:55.379815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:55.462185 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:55.462158    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hs745_c9ac25cf-7824-4fca-8c91-d3c7293e3834/dns-node-resolver/0.log"
Nov 02 17:22:55.559984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:55.559950    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:55.743883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:55.743788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:55.950675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:55.950643    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:56.341769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:56.341736    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:22:56.462392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:56.462367    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqt8d_9867d6a5-3527-4058-80a0-1d8dd1e1590a/node-ca/0.log"
Nov 02 17:22:56.702013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:56.701979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:56.886746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:56.886706    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:57.127212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.127178    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:57.262793 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.262766    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rxx72_1e6b3bff-315b-4fd5-b428-a2da329b0cea/serve-healthcheck-canary/0.log"
Nov 02 17:22:57.313893 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.313863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:57.371694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.371668    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dz5j7"
Nov 02 17:22:57.498081 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.498003    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:57.665651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.665606    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jjlxv_7d3b166c-1f4f-4645-bc93-d76093c412e1/migrator/0.log"
Nov 02 17:22:57.687673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.687622    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:57.972060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:57.972026    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:58.134543 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.134508    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:22:58.135036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.134565    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:22:58.135036 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:58.134712    4476 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:22:58.135036 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:58.134727    4476 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:58.135036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.134756    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:58.135036 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:58.134790    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls podName:19284cc9-6243-40af-862b-dc011abb8b7c nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.134768844 +0000 UTC m=+16.975374774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls") pod "dns-default-n8b9d" (UID: "19284cc9-6243-40af-862b-dc011abb8b7c") : secret "dns-default-metrics-tls" not found
Nov 02 17:22:58.135036 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:22:58.134876    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls podName:9eb3b92e-c809-4e8d-a886-9d3ef6b35da8 nodeName:}" failed. No retries permitted until 2026-11-02 17:23:06.134851684 +0000 UTC m=+16.975457599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pmnsd" (UID: "9eb3b92e-c809-4e8d-a886-9d3ef6b35da8") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:22:58.135295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.135056    4476 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:22:58.137208 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.137193    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00939ab6-259b-42cd-be51-95182c9ba479-metrics-certs\") pod \"network-metrics-daemon-2b8zt\" (UID: \"00939ab6-259b-42cd-be51-95182c9ba479\") " pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:58.160203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.160178    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2b8zt"
Nov 02 17:22:58.181649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.181609    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:58.471666 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.471622    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:22:58.471831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.471820    4476 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:22:58.482824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.482756    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:58.543348 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:22:58.543304    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00939ab6_259b_42cd_be51_95182c9ba479.slice/crio-b4f4e7f11b7e4ae5c3c3af4f6910b716d2b71c13b996bb0ea7283194d8268a59 WatchSource:0}: Error finding container b4f4e7f11b7e4ae5c3c3af4f6910b716d2b71c13b996bb0ea7283194d8268a59: Status 404 returned error can't find the container with id b4f4e7f11b7e4ae5c3c3af4f6910b716d2b71c13b996bb0ea7283194d8268a59
Nov 02 17:22:58.573846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.573818    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2b8zt"]
Nov 02 17:22:58.893608 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:58.893533    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:59.077142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:59.077108    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:59.301968 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:59.301884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:22:59.449998 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:59.449963    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2b8zt" event={"ID":"00939ab6-259b-42cd-be51-95182c9ba479","Type":"ContainerStarted","Data":"b4f4e7f11b7e4ae5c3c3af4f6910b716d2b71c13b996bb0ea7283194d8268a59"}
Nov 02 17:22:59.682555 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:22:59.682522    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:00.455349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:00.455306    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2b8zt" event={"ID":"00939ab6-259b-42cd-be51-95182c9ba479","Type":"ContainerStarted","Data":"5ad425622a99d4b34e609169f256d4a63cbc577f1088638cf702f6dd0bc84960"}
Nov 02 17:23:00.455775 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:00.455367    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2b8zt" event={"ID":"00939ab6-259b-42cd-be51-95182c9ba479","Type":"ContainerStarted","Data":"25e530e73398925a289d9f82c4337bec3c982fbbb57e6dc8b9ae05ca9395c0b9"}
Nov 02 17:23:00.482879 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:00.482846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:00.889997 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:00.889963    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:01.078415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:01.078341    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:01.295340 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:01.295308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:01.690496 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:01.690458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:02.480196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:02.480164    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:02.880806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:02.880779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:03.085736 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:03.085701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:03.281918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:03.281837    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:03.690243 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:03.690201    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:04.493926 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:04.493893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:04.894177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:04.894142    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:05.081979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:05.081950    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:05.280861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:05.280786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:05.476666 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:05.476615    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:06.199670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.199615    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:23:06.200060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.199697    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:23:06.202287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.202267    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9eb3b92e-c809-4e8d-a886-9d3ef6b35da8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pmnsd\" (UID: \"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8\") " pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:23:06.202395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.202371    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19284cc9-6243-40af-862b-dc011abb8b7c-metrics-tls\") pod \"dns-default-n8b9d\" (UID: \"19284cc9-6243-40af-862b-dc011abb8b7c\") " pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:23:06.264234 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.264201    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:23:06.265295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.265277    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pmnsd"
Nov 02 17:23:06.691297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.691206    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8b9d"]
Nov 02 17:23:06.695711 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:06.695688    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19284cc9_6243_40af_862b_dc011abb8b7c.slice/crio-39de5039a7fbb609712c92cc8eb2f147e0879cd7e7a418c49a4041f324d18a98 WatchSource:0}: Error finding container 39de5039a7fbb609712c92cc8eb2f147e0879cd7e7a418c49a4041f324d18a98: Status 404 returned error can't find the container with id 39de5039a7fbb609712c92cc8eb2f147e0879cd7e7a418c49a4041f324d18a98
Nov 02 17:23:06.742710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:06.742676    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pmnsd"]
Nov 02 17:23:06.746809 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:06.746781    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb3b92e_c809_4e8d_a886_9d3ef6b35da8.slice/crio-b7ba87727ae9f0a81a4667c3bd2656ec0463eb8194c5e11dc35eb7964576bdcc WatchSource:0}: Error finding container b7ba87727ae9f0a81a4667c3bd2656ec0463eb8194c5e11dc35eb7964576bdcc: Status 404 returned error can't find the container with id b7ba87727ae9f0a81a4667c3bd2656ec0463eb8194c5e11dc35eb7964576bdcc
Nov 02 17:23:07.137767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.137734    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-554f9b965b-7n8kt"]
Nov 02 17:23:07.139591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.139575    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.183492 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.183459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:07.206453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206414    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-registry-tls\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206459    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-trusted-ca\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206552    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-ca-trust-extracted\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206581    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-registry-certificates\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206707    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-image-registry-private-configuration\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206729    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-bound-sa-token\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206752    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-installation-pull-secrets\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.206837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.206789    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj5z\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-kube-api-access-brj5z\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.276093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.276057    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 17:23:07.276093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.276076    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xcdtc\""
Nov 02 17:23:07.276093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.276089    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 17:23:07.276333 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.276097    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 17:23:07.291605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.291574    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 17:23:07.291933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.291909    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554f9b965b-7n8kt"]
Nov 02 17:23:07.307323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307294    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-ca-trust-extracted\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307326    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-registry-certificates\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307501 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307362    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-image-registry-private-configuration\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307501 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307479    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-bound-sa-token\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307514    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-installation-pull-secrets\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307532    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj5z\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-kube-api-access-brj5z\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307588    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-registry-tls\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307787 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307612    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-trusted-ca\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.307787 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.307766    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-ca-trust-extracted\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.308421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.308315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-registry-certificates\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.308957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.308931    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-trusted-ca\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.310435 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.310411    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-image-registry-private-configuration\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.310534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.310446    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-registry-tls\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.310856 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.310840    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-installation-pull-secrets\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.366365 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.366338    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj5z\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-kube-api-access-brj5z\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.366515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.366387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3-bound-sa-token\") pod \"image-registry-554f9b965b-7n8kt\" (UID: \"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3\") " pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.370825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.370795    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:07.449392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.449293    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:07.480698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.480662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8b9d" event={"ID":"19284cc9-6243-40af-862b-dc011abb8b7c","Type":"ContainerStarted","Data":"39de5039a7fbb609712c92cc8eb2f147e0879cd7e7a418c49a4041f324d18a98"}
Nov 02 17:23:07.482431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.482403    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pmnsd" event={"ID":"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8","Type":"ContainerStarted","Data":"bff7b5d65b36b9e91034c718d3087baf6b3a962681e8e33cb3379f74393700e0"}
Nov 02 17:23:07.482548 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.482440    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pmnsd" event={"ID":"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8","Type":"ContainerStarted","Data":"b7ba87727ae9f0a81a4667c3bd2656ec0463eb8194c5e11dc35eb7964576bdcc"}
Nov 02 17:23:07.789664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.789563    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:07.876721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.876692    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-554f9b965b-7n8kt"]
Nov 02 17:23:07.880872 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:07.880840    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ce9cc1_355e_4bc7_9629_9d5a4a5e1df3.slice/crio-c5d44f35c354b10b5d70b72e8039ff3ac8392d4dc0dc013e31f75d1f4237f60d WatchSource:0}: Error finding container c5d44f35c354b10b5d70b72e8039ff3ac8392d4dc0dc013e31f75d1f4237f60d: Status 404 returned error can't find the container with id c5d44f35c354b10b5d70b72e8039ff3ac8392d4dc0dc013e31f75d1f4237f60d
Nov 02 17:23:07.973534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:07.973502    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:08.489151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:08.489114    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt" event={"ID":"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3","Type":"ContainerStarted","Data":"33b60c5309483f46cfff31f2a82a88c0f57e75de29278d936a83324a4ed7917d"}
Nov 02 17:23:08.489610 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:08.489172    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt" event={"ID":"e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3","Type":"ContainerStarted","Data":"c5d44f35c354b10b5d70b72e8039ff3ac8392d4dc0dc013e31f75d1f4237f60d"}
Nov 02 17:23:08.489610 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:08.489247    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:08.491238 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:08.491212    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pmnsd" event={"ID":"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8","Type":"ContainerStarted","Data":"f8e72398b03754df2153277929eca3bd6846ce346ab3108c0b5c56d5d9125ef9"}
Nov 02 17:23:08.606297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:08.606230    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt" podStartSLOduration=1.606211683 podStartE2EDuration="1.606211683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:08.60544569 +0000 UTC m=+19.446051629" watchObservedRunningTime="2026-11-02 17:23:08.606211683 +0000 UTC m=+19.446817645"
Nov 02 17:23:09.496910 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:09.496874    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8b9d" event={"ID":"19284cc9-6243-40af-862b-dc011abb8b7c","Type":"ContainerStarted","Data":"9fd2f2893e1f766251ef9a076cff444aad1f5a40286a1463c919efc1c589f907"}
Nov 02 17:23:09.497358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:09.496935    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8b9d" event={"ID":"19284cc9-6243-40af-862b-dc011abb8b7c","Type":"ContainerStarted","Data":"39d0a0d1d052d678d54d4bc1bc0b4b066634ea542b9c133b2f405281e4f866c6"}
Nov 02 17:23:10.502190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:10.502146    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pmnsd" event={"ID":"9eb3b92e-c809-4e8d-a886-9d3ef6b35da8","Type":"ContainerStarted","Data":"653a0c0e71ddbb4806343df50366a5b2c1e4a4e197da51b6c267f27816382070"}
Nov 02 17:23:10.502693 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:10.502320    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:23:11.570755 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:11.570719    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:11.800486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:11.800447    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:11.983096 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:11.983064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:12.666918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:12.666884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:12.855610 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:12.855556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:15.743420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:15.743393    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhl5c"
Nov 02 17:23:16.923046 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:16.923011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:17.104716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:17.104682    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:17.316924 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:17.316844    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:17.535017 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:17.534985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:19.229199 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:19.229170    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:19.417493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:19.417461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:20.507986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:20.507953    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n8b9d"
Nov 02 17:23:22.363873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:22.363842    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:23.181282 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:23.181248    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:23.437221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:23.437145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:23.645080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:23.645039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:23.864147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:23.864115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:23.886196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:23.886163    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-45wxf"]
Nov 02 17:23:23.918831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:23.918803    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019180    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-tls\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019219    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-metrics-client-ca\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019251    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7bbl\" (UniqueName: \"kubernetes.io/projected/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-kube-api-access-w7bbl\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019300    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-sys\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019403    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019483    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-wtmp\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019517    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-textfile\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019564    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.019605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.019586    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-root\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.071529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.071495    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:24.090960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.090931    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 17:23:24.090960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.090946    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 17:23:24.091414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.091396    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 17:23:24.091414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.091408    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 17:23:24.091538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.091438    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv5xv\""
Nov 02 17:23:24.091538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.091439    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 17:23:24.091664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.091588    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 17:23:24.120296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120234    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-sys\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120275    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120298    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-wtmp\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120317    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-textfile\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120332    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120357    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-root\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120365    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-sys\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120386    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-tls\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120423    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-metrics-client-ca\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120451    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-wtmp\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120462    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7bbl\" (UniqueName: \"kubernetes.io/projected/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-kube-api-access-w7bbl\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120865 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120478    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-root\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120865 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120750    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-textfile\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.120971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120958    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.121026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.120992    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-metrics-client-ca\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.122978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.122949    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.123077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.123031    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-node-exporter-tls\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.176984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.176956    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7bbl\" (UniqueName: \"kubernetes.io/projected/a778c105-00fa-4d5e-a614-2a12c6c5b9e7-kube-api-access-w7bbl\") pod \"node-exporter-45wxf\" (UID: \"a778c105-00fa-4d5e-a614-2a12c6c5b9e7\") " pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.271746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.271709    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-45wxf"
Nov 02 17:23:24.306121 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:24.306097    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda778c105_00fa_4d5e_a614_2a12c6c5b9e7.slice/crio-11efcace8acf262e33b15468e90c8f18356e5689517ae9864632410f68ee8be4 WatchSource:0}: Error finding container 11efcace8acf262e33b15468e90c8f18356e5689517ae9864632410f68ee8be4: Status 404 returned error can't find the container with id 11efcace8acf262e33b15468e90c8f18356e5689517ae9864632410f68ee8be4
Nov 02 17:23:24.373695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.373610    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:24.548598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.548560    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-45wxf" event={"ID":"a778c105-00fa-4d5e-a614-2a12c6c5b9e7","Type":"ContainerStarted","Data":"11efcace8acf262e33b15468e90c8f18356e5689517ae9864632410f68ee8be4"}
Nov 02 17:23:24.583929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:24.583895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:25.331609 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:25.331550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:25.535416 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:25.535381    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:25.868188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:25.868153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:26.062804 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.062774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:26.263854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.263821    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:26.344621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.344590    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-slw2l"
Nov 02 17:23:26.561319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.561241    4476 generic.go:356] "Generic (PLEG): container finished" podID="a778c105-00fa-4d5e-a614-2a12c6c5b9e7" containerID="04a5be5e3884b3e1418a306a34132bbaf5ae1150fa6ee672bcdcb4f14e463a72" exitCode=0
Nov 02 17:23:26.561319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.561315    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-45wxf" event={"ID":"a778c105-00fa-4d5e-a614-2a12c6c5b9e7","Type":"ContainerDied","Data":"04a5be5e3884b3e1418a306a34132bbaf5ae1150fa6ee672bcdcb4f14e463a72"}
Nov 02 17:23:26.616678 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.616652    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bd567549c-cbcbb"]
Nov 02 17:23:26.618909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.618892    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.671046 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.671023    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 17:23:26.671146 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.671024    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 17:23:26.671146 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.671060    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 17:23:26.684305 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.684274    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bd567549c-cbcbb"]
Nov 02 17:23:26.743131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743103    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743134    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17bfcc67-9352-4d82-897a-86a6c72cbf0c-metrics-client-ca\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743160    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxzg\" (UniqueName: \"kubernetes.io/projected/17bfcc67-9352-4d82-897a-86a6c72cbf0c-kube-api-access-sdxzg\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743289    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bfcc67-9352-4d82-897a-86a6c72cbf0c-tmp\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743322    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-grpc-tls\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743354    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743419    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-tls\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743452    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.743536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.743478    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.771389 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.771357    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 17:23:26.771532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.771355    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 17:23:26.771532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.771414    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3j43dl4uqb15j\""
Nov 02 17:23:26.771532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.771430    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2ghfk\""
Nov 02 17:23:26.844025 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.843934    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bfcc67-9352-4d82-897a-86a6c72cbf0c-tmp\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844025 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.843976    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-grpc-tls\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844025 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844001    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844030    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-tls\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844085    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844131    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844177    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844203    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17bfcc67-9352-4d82-897a-86a6c72cbf0c-metrics-client-ca\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844241    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxzg\" (UniqueName: \"kubernetes.io/projected/17bfcc67-9352-4d82-897a-86a6c72cbf0c-kube-api-access-sdxzg\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.844586 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.844433    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17bfcc67-9352-4d82-897a-86a6c72cbf0c-tmp\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.845135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.845101    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17bfcc67-9352-4d82-897a-86a6c72cbf0c-metrics-client-ca\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.847357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.847336    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-tls\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.847468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.847376    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.847468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.847437    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.847584 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.847567    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.847743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.847729    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-grpc-tls\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.847990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.847968    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/17bfcc67-9352-4d82-897a-86a6c72cbf0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.900897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.900862    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxzg\" (UniqueName: \"kubernetes.io/projected/17bfcc67-9352-4d82-897a-86a6c72cbf0c-kube-api-access-sdxzg\") pod \"thanos-querier-bd567549c-cbcbb\" (UID: \"17bfcc67-9352-4d82-897a-86a6c72cbf0c\") " pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:26.941766 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:26.941735    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:27.365504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.365467    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bd567549c-cbcbb"]
Nov 02 17:23:27.369096 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:27.369070    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bfcc67_9352_4d82_897a_86a6c72cbf0c.slice/crio-3ac5e430c3860aa3044d8ffb99c078c09e6732e30d7a29640b60eaa183bb1189 WatchSource:0}: Error finding container 3ac5e430c3860aa3044d8ffb99c078c09e6732e30d7a29640b60eaa183bb1189: Status 404 returned error can't find the container with id 3ac5e430c3860aa3044d8ffb99c078c09e6732e30d7a29640b60eaa183bb1189
Nov 02 17:23:27.453930 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.453893    4476 patch_prober.go:29] interesting pod/image-registry-554f9b965b-7n8kt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 17:23:27.454120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.453953    4476 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt" podUID="e1ce9cc1-355e-4bc7-9629-9d5a4a5e1df3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:23:27.567656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.567602    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-45wxf" event={"ID":"a778c105-00fa-4d5e-a614-2a12c6c5b9e7","Type":"ContainerStarted","Data":"b245c69358015ccae237ba876e177a5641816fa194cf95dc23a2cacb60ad5d6a"}
Nov 02 17:23:27.567838 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.567684    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-45wxf" event={"ID":"a778c105-00fa-4d5e-a614-2a12c6c5b9e7","Type":"ContainerStarted","Data":"a4dcb4aa613a341f52423a45221fe28a59ff88e89894fa9ff69bf0bc01ff90cc"}
Nov 02 17:23:27.568918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.568898    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"3ac5e430c3860aa3044d8ffb99c078c09e6732e30d7a29640b60eaa183bb1189"}
Nov 02 17:23:27.688162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:27.688049    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-45wxf" podStartSLOduration=2.705606204 podStartE2EDuration="4.688033393s" totalImagesPullingTime="1.895005407s" totalInitContainerRuntime="87.421782ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:27.687183647 +0000 UTC m=+38.527789604" watchObservedRunningTime="2026-11-02 17:23:27.688033393 +0000 UTC m=+38.528639392"
Nov 02 17:23:28.024494 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:28.024408    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:28.220419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:28.220384    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:28.427247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:28.427211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:28.619968 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:28.619923    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:28.827321 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:28.827233    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:29.065289 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:23:29.065240    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:23:29.386862 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:29.386828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:29.501858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:29.501826    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-554f9b965b-7n8kt"
Nov 02 17:23:29.591621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:29.591580    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:29.804231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:29.804206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:30.013042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:30.012963    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:30.238837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:30.238806    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:30.469433 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:30.469361    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:30.581901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:30.581857    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"f8901834a909c89aa516b152e0ef12e1bcfb593f4e9856fcb2ece2d70309cdaa"}
Nov 02 17:23:30.582042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:30.581925    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"b934ec9c9906763d7f1656f49ad349ebc38878c5a3f59b8163819d26670db59f"}
Nov 02 17:23:30.582042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:30.581952    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"d2090ffb6e965ce19a37e3cd4d2ece5de8a0ced28e29317664ac3b2be6163d2d"}
Nov 02 17:23:31.176794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.176759    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:31.405133 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.405102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:31.588339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.588248    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"aad87c4f2c17f8f6b96811d88fe6a981bc2443f94ba8ac918874819d55db13d9"}
Nov 02 17:23:31.588339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.588307    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"f45d4d5e1f3f22edf4c388ae5eb928d2d6296d43eab741e34174d16b324ce272"}
Nov 02 17:23:31.588339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.588333    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" event={"ID":"17bfcc67-9352-4d82-897a-86a6c72cbf0c","Type":"ContainerStarted","Data":"3014d895ee43cca1342f4437cf0674115abf123ddc66e54dd32d84418c47269d"}
Nov 02 17:23:31.588681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.588397    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:31.859136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:31.859036    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb" podStartSLOduration=2.25210879 podStartE2EDuration="5.859021953s" totalImagesPullingTime="3.606913163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:31.808384569 +0000 UTC m=+42.648990513" watchObservedRunningTime="2026-11-02 17:23:31.859021953 +0000 UTC m=+42.699627929"
Nov 02 17:23:33.036795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:33.036762    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:34.248104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:34.248064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:34.455211 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:34.455169    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:37.599237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:37.599207    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bd567549c-cbcbb"
Nov 02 17:23:38.851844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:38.851805    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:39.067049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:39.067012    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:39.784012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:39.783974    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:40.035009 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.034928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:40.303035 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.302959    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:40.388693 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.388652    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d4d9db5f5-65j9f"]
Nov 02 17:23:40.391371 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.391347    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.443485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.443455    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 17:23:40.453157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453127    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.453332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453166    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.453332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453243    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.453332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453285    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4pg\" (UniqueName: \"kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.453442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453383    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.453442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453425    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.453525 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.453453    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.493124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493095    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 17:23:40.493323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493221    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 17:23:40.493323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493235    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 17:23:40.493323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493252    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 17:23:40.493323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493270    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zjs7h\""
Nov 02 17:23:40.493323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493256    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 17:23:40.493323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.493289    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 17:23:40.494511 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.494491    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4d9db5f5-65j9f"]
Nov 02 17:23:40.504261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.504240    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:40.555529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.554581    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.555529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.554684    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.557695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.556933    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.557695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.556988    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4pg\" (UniqueName: \"kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.557695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.557030    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.557695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.557079    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.557695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.557109    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.560198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.558761    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.560198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.559112    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.560198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.559136    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.560564 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.560543    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.561060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.561040    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.561300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.561283    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.616906 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.616878    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4pg\" (UniqueName: \"kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg\") pod \"console-d4d9db5f5-65j9f\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") " pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.702349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.702314    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:40.804921 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:40.804886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:41.127305 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:41.127269    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4d9db5f5-65j9f"]
Nov 02 17:23:41.131155 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:41.131123    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e32f832_0c84_4985_a602_f9097344054a.slice/crio-a7fe80f32dca428359ce20a82b282ab74171994438bd955bce212d389b149331 WatchSource:0}: Error finding container a7fe80f32dca428359ce20a82b282ab74171994438bd955bce212d389b149331: Status 404 returned error can't find the container with id a7fe80f32dca428359ce20a82b282ab74171994438bd955bce212d389b149331
Nov 02 17:23:41.622754 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:41.622722    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d9db5f5-65j9f" event={"ID":"2e32f832-0c84-4985-a602-f9097344054a","Type":"ContainerStarted","Data":"a7fe80f32dca428359ce20a82b282ab74171994438bd955bce212d389b149331"}
Nov 02 17:23:42.197497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:42.197459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:42.420469 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:42.420435    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:44.427532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:44.427499    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:44.616272 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:44.616232    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:44.634868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:44.634837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d9db5f5-65j9f" event={"ID":"2e32f832-0c84-4985-a602-f9097344054a","Type":"ContainerStarted","Data":"e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960"}
Nov 02 17:23:44.750183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:44.750130    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d4d9db5f5-65j9f" podStartSLOduration=1.682710539 podStartE2EDuration="4.750116663s" totalImagesPullingTime="3.067406124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:44.749300229 +0000 UTC m=+55.589906162" watchObservedRunningTime="2026-11-02 17:23:44.750116663 +0000 UTC m=+55.590722596"
Nov 02 17:23:44.811966 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:44.811929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:49.293492 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.293461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:49.517471 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.517438    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:49.851394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.851365    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9776b7bf6-qjh5p"]
Nov 02 17:23:49.854247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.854230    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.926961    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927003    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927028    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927083    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6c5l\" (UniqueName: \"kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927167    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927198    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927217    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:49.927337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:49.927310    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.027993    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028045    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028073    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028107    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028157    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028187    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028221    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028546 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028242    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6c5l\" (UniqueName: \"kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028696    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.028959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028904    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.029109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.028989    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.029186 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.029165    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.030743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.030723    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.030884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.030864    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.055393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.055359    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 17:23:50.056404 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.056382    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9776b7bf6-qjh5p"]
Nov 02 17:23:50.060348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.060330    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.085247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.085215    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6c5l\" (UniqueName: \"kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l\") pod \"console-9776b7bf6-qjh5p\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") " pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.164402 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.164353    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:23:50.535286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.535202    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9776b7bf6-qjh5p"]
Nov 02 17:23:50.539332 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:23:50.539308    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f7e82a_1332_46ed_987e_105ddedfd835.slice/crio-7675b08252322b0ef1423ee47fe65a66841b56674db3ff4e9ea37dc4e1687829 WatchSource:0}: Error finding container 7675b08252322b0ef1423ee47fe65a66841b56674db3ff4e9ea37dc4e1687829: Status 404 returned error can't find the container with id 7675b08252322b0ef1423ee47fe65a66841b56674db3ff4e9ea37dc4e1687829
Nov 02 17:23:50.658153 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.658106    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9776b7bf6-qjh5p" event={"ID":"70f7e82a-1332-46ed-987e-105ddedfd835","Type":"ContainerStarted","Data":"b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92"}
Nov 02 17:23:50.658336 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.658177    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9776b7bf6-qjh5p" event={"ID":"70f7e82a-1332-46ed-987e-105ddedfd835","Type":"ContainerStarted","Data":"7675b08252322b0ef1423ee47fe65a66841b56674db3ff4e9ea37dc4e1687829"}
Nov 02 17:23:50.702930 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.702894    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:50.703119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.702946    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:50.708093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.708070    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:50.773381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:50.773321    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9776b7bf6-qjh5p" podStartSLOduration=1.7733065030000001 podStartE2EDuration="1.773306503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:50.772762006 +0000 UTC m=+61.613367940" watchObservedRunningTime="2026-11-02 17:23:50.773306503 +0000 UTC m=+61.613912435"
Nov 02 17:23:51.665443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:51.665415    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:23:55.417552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:55.417523    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:55.609743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:55.609713    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:58.616292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:58.616254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:58.803409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:23:58.803371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:23:59.058580 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:23:59.058491    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:24:00.164719 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:00.164673    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:24:00.165236 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:00.164772    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:24:00.169556 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:00.169532    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:24:00.695725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:00.695699    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:24:00.866180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:00.866147    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4d9db5f5-65j9f"]
Nov 02 17:24:04.634404 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:04.634331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:04.875182 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:04.875152    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:11.371935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.371900    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-kxkns-mjrds"]
Nov 02 17:24:11.372378 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.372122    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-kxkns-mjrds" podUID="df1447be-4ab8-40c9-8606-de70d0920e65" containerName="insights-gathering" containerID="cri-o://aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8" gracePeriod=30
Nov 02 17:24:11.625064 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.625004    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:24:11.705386 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705350    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") pod \"df1447be-4ab8-40c9-8606-de70d0920e65\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") "
Nov 02 17:24:11.705683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705402    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") pod \"df1447be-4ab8-40c9-8606-de70d0920e65\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") "
Nov 02 17:24:11.705683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705513    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") pod \"df1447be-4ab8-40c9-8606-de70d0920e65\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") "
Nov 02 17:24:11.705683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705548    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") pod \"df1447be-4ab8-40c9-8606-de70d0920e65\" (UID: \"df1447be-4ab8-40c9-8606-de70d0920e65\") "
Nov 02 17:24:11.705879 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705777    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path" pod "df1447be-4ab8-40c9-8606-de70d0920e65" (UID: "df1447be-4ab8-40c9-8606-de70d0920e65"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:24:11.705879 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705838    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle" pod "df1447be-4ab8-40c9-8606-de70d0920e65" (UID: "df1447be-4ab8-40c9-8606-de70d0920e65"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:24:11.705879 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.705847    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle" pod "df1447be-4ab8-40c9-8606-de70d0920e65" (UID: "df1447be-4ab8-40c9-8606-de70d0920e65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:24:11.708180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.708154    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf" pod "df1447be-4ab8-40c9-8606-de70d0920e65" (UID: "df1447be-4ab8-40c9-8606-de70d0920e65"). InnerVolumeSpecName "kube-api-access-h5vdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:24:11.730091 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.730056    4476 generic.go:356] "Generic (PLEG): container finished" podID="df1447be-4ab8-40c9-8606-de70d0920e65" containerID="aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8" exitCode=2
Nov 02 17:24:11.730217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.730096    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kxkns-mjrds" event={"ID":"df1447be-4ab8-40c9-8606-de70d0920e65","Type":"ContainerDied","Data":"aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8"}
Nov 02 17:24:11.730217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.730152    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kxkns-mjrds" event={"ID":"df1447be-4ab8-40c9-8606-de70d0920e65","Type":"ContainerDied","Data":"33504445c6b2236fb56451c395bfcab78bce2ea608962936d53a1af17432af1c"}
Nov 02 17:24:11.730217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.730108    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kxkns-mjrds"
Nov 02 17:24:11.730369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.730184    4476 scope.go:122] "RemoveContainer" containerID="aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8"
Nov 02 17:24:11.738525 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.738510    4476 scope.go:122] "RemoveContainer" containerID="aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8"
Nov 02 17:24:11.738852 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:24:11.738824    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8\": container with ID starting with aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8 not found: ID does not exist" containerID="aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8"
Nov 02 17:24:11.738918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.738861    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8"} err="failed to get container status \"aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8\": rpc error: code = NotFound desc = could not find container \"aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8\": container with ID starting with aa858ceda7cdd72191bad05e7fdbf8495c8b7e1bf8dfa1e71424ccdcd2740bb8 not found: ID does not exist"
Nov 02 17:24:11.807097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.807069    4476 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/df1447be-4ab8-40c9-8606-de70d0920e65-archives-path\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:11.807097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.807093    4476 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-service-ca-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:11.807263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.807104    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vdf\" (UniqueName: \"kubernetes.io/projected/df1447be-4ab8-40c9-8606-de70d0920e65-kube-api-access-h5vdf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:11.807263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.807117    4476 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1447be-4ab8-40c9-8606-de70d0920e65-trusted-ca-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:11.881801 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.881726    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-kxkns-mjrds"]
Nov 02 17:24:11.887083 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:11.887060    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-kxkns-mjrds"]
Nov 02 17:24:13.380240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:13.380202    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1447be-4ab8-40c9-8606-de70d0920e65" path="/var/lib/kubelet/pods/df1447be-4ab8-40c9-8606-de70d0920e65/volumes"
Nov 02 17:24:17.677826 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:17.677790    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:17.885173 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:17.885136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:18.112065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:18.112002    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:18.315117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:18.315073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:19.237944 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:19.237894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:19.424449 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:19.424414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:25.886361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:25.886296    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d4d9db5f5-65j9f" podUID="2e32f832-0c84-4985-a602-f9097344054a" containerName="console" containerID="cri-o://e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960" gracePeriod=15
Nov 02 17:24:26.160857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.160834    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:24:26.325059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325028    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325094    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325113    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325139    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325165    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg\" (UniqueName: \"kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325195    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325463 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325227    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert\") pod \"2e32f832-0c84-4985-a602-f9097344054a\" (UID: \"2e32f832-0c84-4985-a602-f9097344054a\") "
Nov 02 17:24:26.325523 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325479    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:24:26.325687 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325653    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:24:26.325784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325719    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:24:26.325784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.325744    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:24:26.327521 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.327499    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:24:26.327615 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.327527    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:24:26.327672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.327649    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg" pod "2e32f832-0c84-4985-a602-f9097344054a" (UID: "2e32f832-0c84-4985-a602-f9097344054a"). InnerVolumeSpecName "kube-api-access-ln4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426004    4476 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-serving-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426031    4476 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e32f832-0c84-4985-a602-f9097344054a-console-oauth-config\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426042    4476 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426050    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4pg\" (UniqueName: \"kubernetes.io/projected/2e32f832-0c84-4985-a602-f9097344054a-kube-api-access-ln4pg\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426060    4476 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e32f832-0c84-4985-a602-f9097344054a-tmp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426070    4476 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-oauth-serving-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.426078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.426079    4476 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e32f832-0c84-4985-a602-f9097344054a-console-config\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:24:26.779361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.779277    4476 generic.go:356] "Generic (PLEG): container finished" podID="2e32f832-0c84-4985-a602-f9097344054a" containerID="e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960" exitCode=0
Nov 02 17:24:26.779361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.779326    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4d9db5f5-65j9f"
Nov 02 17:24:26.779597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.779370    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d9db5f5-65j9f" event={"ID":"2e32f832-0c84-4985-a602-f9097344054a","Type":"ContainerDied","Data":"e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960"}
Nov 02 17:24:26.779597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.779424    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4d9db5f5-65j9f" event={"ID":"2e32f832-0c84-4985-a602-f9097344054a","Type":"ContainerDied","Data":"a7fe80f32dca428359ce20a82b282ab74171994438bd955bce212d389b149331"}
Nov 02 17:24:26.779597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.779452    4476 scope.go:122] "RemoveContainer" containerID="e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960"
Nov 02 17:24:26.787781 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.787765    4476 scope.go:122] "RemoveContainer" containerID="e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960"
Nov 02 17:24:26.788020 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:24:26.788004    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960\": container with ID starting with e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960 not found: ID does not exist" containerID="e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960"
Nov 02 17:24:26.788063 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.788028    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960"} err="failed to get container status \"e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960\": rpc error: code = NotFound desc = could not find container \"e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960\": container with ID starting with e8e73a665519426ecd8367a182d64e7fa359c4160664ffdf7ccec69fd876e960 not found: ID does not exist"
Nov 02 17:24:26.942524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.942493    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4d9db5f5-65j9f"]
Nov 02 17:24:26.991643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:26.991602    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d4d9db5f5-65j9f"]
Nov 02 17:24:27.380024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:27.379995    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e32f832-0c84-4985-a602-f9097344054a" path="/var/lib/kubelet/pods/2e32f832-0c84-4985-a602-f9097344054a/volumes"
Nov 02 17:24:29.027214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027184    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d9d456f7-55dqm"]
Nov 02 17:24:29.027598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027454    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2e32f832-0c84-4985-a602-f9097344054a" containerName="console"
Nov 02 17:24:29.027598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027465    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e32f832-0c84-4985-a602-f9097344054a" containerName="console"
Nov 02 17:24:29.027598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027485    4476 cpu_manager.go:414] "containerMap: removing container" podUID="df1447be-4ab8-40c9-8606-de70d0920e65" containerName="insights-gathering"
Nov 02 17:24:29.027598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027490    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1447be-4ab8-40c9-8606-de70d0920e65" containerName="insights-gathering"
Nov 02 17:24:29.027598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027528    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1447be-4ab8-40c9-8606-de70d0920e65" containerName="insights-gathering"
Nov 02 17:24:29.027598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.027537    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e32f832-0c84-4985-a602-f9097344054a" containerName="console"
Nov 02 17:24:29.030050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.030035    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.057330 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:24:29.057303    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:24:29.087672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.087649    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d9d456f7-55dqm"]
Nov 02 17:24:29.143433 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143405    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143534    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143581    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143608    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknw4\" (UniqueName: \"kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143740    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143871 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143826    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143924 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143874    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.143977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.143946    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245208    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245271    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245295    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245317    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknw4\" (UniqueName: \"kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245346    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245385    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245426    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245455    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.245786 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.245598    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.246093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.246037    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.246322 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.246151    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.246390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.246354    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.246736 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.246706    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.247961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.247943    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.248108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.248092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.301770 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.301715    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknw4\" (UniqueName: \"kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4\") pod \"console-84d9d456f7-55dqm\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") " pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.341101 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.341078    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:29.347849 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.347828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:29.720768 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.720740    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d9d456f7-55dqm"]
Nov 02 17:24:29.752840 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:24:29.752813    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd2abe2_c5d5_4ffb_a0a0_d14c79165ecc.slice/crio-f877b6be0b733c35108ebdfe72c6d140d07890e6fa99f7e4a9d3d247cf2108c2 WatchSource:0}: Error finding container f877b6be0b733c35108ebdfe72c6d140d07890e6fa99f7e4a9d3d247cf2108c2: Status 404 returned error can't find the container with id f877b6be0b733c35108ebdfe72c6d140d07890e6fa99f7e4a9d3d247cf2108c2
Nov 02 17:24:29.792222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:29.792196    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9d456f7-55dqm" event={"ID":"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc","Type":"ContainerStarted","Data":"f877b6be0b733c35108ebdfe72c6d140d07890e6fa99f7e4a9d3d247cf2108c2"}
Nov 02 17:24:30.756323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:30.756289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:30.796253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:30.796219    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9d456f7-55dqm" event={"ID":"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc","Type":"ContainerStarted","Data":"416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9"}
Nov 02 17:24:30.913606 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:30.913560    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84d9d456f7-55dqm" podStartSLOduration=2.913546111 podStartE2EDuration="2.913546111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:24:30.912883353 +0000 UTC m=+101.753489328" watchObservedRunningTime="2026-11-02 17:24:30.913546111 +0000 UTC m=+101.754152044"
Nov 02 17:24:31.026481 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:31.026367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:39.341421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:39.341372    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:39.341951 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:39.341450    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:39.345883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:39.345863    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:39.828536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:39.828505    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:24:39.993977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:39.993947    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9776b7bf6-qjh5p"]
Nov 02 17:24:40.104937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:40.104836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:42.160849 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:42.160820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:55.556873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:55.556831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:24:59.056847 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:24:59.056812    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:24:59.357107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:24:59.357028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:02.937386 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.937308    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-574c4f5976-q8k8k"]
Nov 02 17:25:02.940393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.940377    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985553    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-console-config\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985584    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57nlp\" (UniqueName: \"kubernetes.io/projected/951b8982-246b-4511-8067-d47986ac90e3-kube-api-access-57nlp\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985610    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/951b8982-246b-4511-8067-d47986ac90e3-console-serving-cert\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985706    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-trusted-ca-bundle\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985752    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-service-ca\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985801    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/951b8982-246b-4511-8067-d47986ac90e3-console-oauth-config\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.985859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985841    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/951b8982-246b-4511-8067-d47986ac90e3-tmp\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:02.986011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:02.985943    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-oauth-serving-cert\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.042094 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.042061    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574c4f5976-q8k8k"]
Nov 02 17:25:03.087084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087059    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-console-config\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087090    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57nlp\" (UniqueName: \"kubernetes.io/projected/951b8982-246b-4511-8067-d47986ac90e3-kube-api-access-57nlp\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087118    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/951b8982-246b-4511-8067-d47986ac90e3-console-serving-cert\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087148    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-trusted-ca-bundle\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087352    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-service-ca\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087391    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/951b8982-246b-4511-8067-d47986ac90e3-console-oauth-config\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087419    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/951b8982-246b-4511-8067-d47986ac90e3-tmp\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087455    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-oauth-serving-cert\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.087919 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087895    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/951b8982-246b-4511-8067-d47986ac90e3-tmp\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.088031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.087904    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-console-config\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.088140 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.088119    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-oauth-serving-cert\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.088226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.088211    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-trusted-ca-bundle\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.088450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.088429    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/951b8982-246b-4511-8067-d47986ac90e3-service-ca\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.089748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.089729    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/951b8982-246b-4511-8067-d47986ac90e3-console-serving-cert\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.089887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.089870    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/951b8982-246b-4511-8067-d47986ac90e3-console-oauth-config\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.149959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.149936    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57nlp\" (UniqueName: \"kubernetes.io/projected/951b8982-246b-4511-8067-d47986ac90e3-kube-api-access-57nlp\") pod \"console-574c4f5976-q8k8k\" (UID: \"951b8982-246b-4511-8067-d47986ac90e3\") " pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.251504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.251434    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:03.260113 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.260074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:03.457958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.457927    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:03.671446 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.671415    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574c4f5976-q8k8k"]
Nov 02 17:25:03.676009 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:25:03.675984    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951b8982_246b_4511_8067_d47986ac90e3.slice/crio-232bf699f101dbd2785edebf298ba69d8e004e51ab30e98f07287a62482076a3 WatchSource:0}: Error finding container 232bf699f101dbd2785edebf298ba69d8e004e51ab30e98f07287a62482076a3: Status 404 returned error can't find the container with id 232bf699f101dbd2785edebf298ba69d8e004e51ab30e98f07287a62482076a3
Nov 02 17:25:03.718869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.718846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:03.902385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.902351    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:03.903121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.903093    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c4f5976-q8k8k" event={"ID":"951b8982-246b-4511-8067-d47986ac90e3","Type":"ContainerStarted","Data":"3681d269f30151c5739cb95f3c3209b28be33c6c01ebc0ccf9945415b27698c1"}
Nov 02 17:25:03.903255 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:03.903172    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c4f5976-q8k8k" event={"ID":"951b8982-246b-4511-8067-d47986ac90e3","Type":"ContainerStarted","Data":"232bf699f101dbd2785edebf298ba69d8e004e51ab30e98f07287a62482076a3"}
Nov 02 17:25:04.020646 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:04.020538    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-574c4f5976-q8k8k" podStartSLOduration=2.020524783 podStartE2EDuration="2.020524783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:25:04.019286073 +0000 UTC m=+134.859892019" watchObservedRunningTime="2026-11-02 17:25:04.020524783 +0000 UTC m=+134.861130793"
Nov 02 17:25:05.014069 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.014005    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9776b7bf6-qjh5p" podUID="70f7e82a-1332-46ed-987e-105ddedfd835" containerName="console" containerID="cri-o://b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92" gracePeriod=15
Nov 02 17:25:05.263437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.263410    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:25:05.305785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305735    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.305785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305777    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.305903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305806    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.305903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305835    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.305903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305871    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.306059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305924    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l\" (UniqueName: \"kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.306059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305971    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.306059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.305997    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca\") pod \"70f7e82a-1332-46ed-987e-105ddedfd835\" (UID: \"70f7e82a-1332-46ed-987e-105ddedfd835\") "
Nov 02 17:25:05.306200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306052    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:25:05.306200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306173    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:05.306330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306305    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:05.306390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306352    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:05.306390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306369    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:05.306485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306386    4476 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-console-config\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.306485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306402    4476 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70f7e82a-1332-46ed-987e-105ddedfd835-tmp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.306485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.306412    4476 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-oauth-serving-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.308253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.308225    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:25:05.308253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.308245    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:25:05.308379 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.308267    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l" pod "70f7e82a-1332-46ed-987e-105ddedfd835" (UID: "70f7e82a-1332-46ed-987e-105ddedfd835"). InnerVolumeSpecName "kube-api-access-k6c5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:25:05.406898 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.406870    4476 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-oauth-config\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.406898 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.406895    4476 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70f7e82a-1332-46ed-987e-105ddedfd835-console-serving-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.407055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.406905    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6c5l\" (UniqueName: \"kubernetes.io/projected/70f7e82a-1332-46ed-987e-105ddedfd835-kube-api-access-k6c5l\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.407055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.406917    4476 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-trusted-ca-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.407055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.406926    4476 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70f7e82a-1332-46ed-987e-105ddedfd835-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:05.911118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.911086    4476 generic.go:356] "Generic (PLEG): container finished" podID="70f7e82a-1332-46ed-987e-105ddedfd835" containerID="b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92" exitCode=0
Nov 02 17:25:05.911289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.911132    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9776b7bf6-qjh5p"
Nov 02 17:25:05.911289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.911148    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9776b7bf6-qjh5p" event={"ID":"70f7e82a-1332-46ed-987e-105ddedfd835","Type":"ContainerDied","Data":"b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92"}
Nov 02 17:25:05.911289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.911188    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9776b7bf6-qjh5p" event={"ID":"70f7e82a-1332-46ed-987e-105ddedfd835","Type":"ContainerDied","Data":"7675b08252322b0ef1423ee47fe65a66841b56674db3ff4e9ea37dc4e1687829"}
Nov 02 17:25:05.911289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.911206    4476 scope.go:122] "RemoveContainer" containerID="b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92"
Nov 02 17:25:05.918897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.918876    4476 scope.go:122] "RemoveContainer" containerID="b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92"
Nov 02 17:25:05.919155 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:25:05.919138    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92\": container with ID starting with b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92 not found: ID does not exist" containerID="b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92"
Nov 02 17:25:05.919216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:05.919173    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92"} err="failed to get container status \"b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92\": rpc error: code = NotFound desc = could not find container \"b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92\": container with ID starting with b535485d284fad6f02d20ecad8c67b1d2f79304b695ff3ddf9019a23baa0ef92 not found: ID does not exist"
Nov 02 17:25:06.072921 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:06.072888    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9776b7bf6-qjh5p"]
Nov 02 17:25:06.122033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:06.121999    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9776b7bf6-qjh5p"]
Nov 02 17:25:07.380966 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:07.380931    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f7e82a-1332-46ed-987e-105ddedfd835" path="/var/lib/kubelet/pods/70f7e82a-1332-46ed-987e-105ddedfd835/volumes"
Nov 02 17:25:08.348701 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:08.348668    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:08.556052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:08.556020    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:13.252072 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:13.252034    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:13.252465 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:13.252104    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:13.256679 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:13.256655    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:13.942860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:13.942830    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574c4f5976-q8k8k"
Nov 02 17:25:14.111221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:14.111188    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d9d456f7-55dqm"]
Nov 02 17:25:19.243818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:19.243785    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:19.439877 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:19.439843    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:29.057501 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:25:29.057442    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:25:39.130467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.130421    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84d9d456f7-55dqm" podUID="3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" containerName="console" containerID="cri-o://416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9" gracePeriod=15
Nov 02 17:25:39.377957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.377931    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:25:39.557733 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.557622    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.557733 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.557687    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.557733 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.557713    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.558008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.557909    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.558008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.557953    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.558008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558005    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.558158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558039    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.558158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558050    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:39.558158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558085    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:39.558158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558086    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4\" (UniqueName: \"kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4\") pod \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\" (UID: \"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc\") "
Nov 02 17:25:39.558158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558140    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:39.558404 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558378    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:25:39.558451 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558384    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:25:39.558451 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558445    4476 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-config\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.558536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558459    4476 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-trusted-ca-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.558536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.558470    4476 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-oauth-serving-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.560135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.560113    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:25:39.560614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.560590    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:25:39.560737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.560614    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4" pod "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" (UID: "3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc"). InnerVolumeSpecName "kube-api-access-kknw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:25:39.659791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.659746    4476 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-oauth-config\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.659791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.659783    4476 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-console-serving-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.659791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.659793    4476 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.659791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.659801    4476 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-tmp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:39.660062 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:39.659818    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknw4\" (UniqueName: \"kubernetes.io/projected/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc-kube-api-access-kknw4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:25:40.023172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.023137    4476 generic.go:356] "Generic (PLEG): container finished" podID="3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" containerID="416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9" exitCode=0
Nov 02 17:25:40.023390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.023182    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9d456f7-55dqm" event={"ID":"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc","Type":"ContainerDied","Data":"416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9"}
Nov 02 17:25:40.023390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.023197    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9d456f7-55dqm"
Nov 02 17:25:40.023390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.023264    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9d456f7-55dqm" event={"ID":"3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc","Type":"ContainerDied","Data":"f877b6be0b733c35108ebdfe72c6d140d07890e6fa99f7e4a9d3d247cf2108c2"}
Nov 02 17:25:40.023390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.023291    4476 scope.go:122] "RemoveContainer" containerID="416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9"
Nov 02 17:25:40.031778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.031763    4476 scope.go:122] "RemoveContainer" containerID="416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9"
Nov 02 17:25:40.032047 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:25:40.032026    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9\": container with ID starting with 416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9 not found: ID does not exist" containerID="416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9"
Nov 02 17:25:40.032103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.032060    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9"} err="failed to get container status \"416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9\": rpc error: code = NotFound desc = could not find container \"416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9\": container with ID starting with 416b2fa844a64acb9deb9942fece554fb0f10da9f225b42568871f5ab333cbf9 not found: ID does not exist"
Nov 02 17:25:40.188825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.188790    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d9d456f7-55dqm"]
Nov 02 17:25:40.238327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.238299    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d9d456f7-55dqm"]
Nov 02 17:25:40.462193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:40.462164    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:25:41.379887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:41.379834    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" path="/var/lib/kubelet/pods/3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc/volumes"
Nov 02 17:25:59.057669 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:25:59.057609    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:25:59.523931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:25:59.523885    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:29.057337 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:26:29.057293    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:26:37.147801 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:37.147698    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:37.490118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:37.490008    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:38.822812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.822779    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"]
Nov 02 17:26:38.823177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.823041    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" containerName="console"
Nov 02 17:26:38.823177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.823052    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" containerName="console"
Nov 02 17:26:38.823177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.823060    4476 cpu_manager.go:414] "containerMap: removing container" podUID="70f7e82a-1332-46ed-987e-105ddedfd835" containerName="console"
Nov 02 17:26:38.823177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.823065    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f7e82a-1332-46ed-987e-105ddedfd835" containerName="console"
Nov 02 17:26:38.823177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.823121    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd2abe2-c5d5-4ffb-a0a0-d14c79165ecc" containerName="console"
Nov 02 17:26:38.823177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.823127    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f7e82a-1332-46ed-987e-105ddedfd835" containerName="console"
Nov 02 17:26:38.825870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.825855    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.870225 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.870195    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.870363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.870235    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.870363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.870260    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrx2\" (UniqueName: \"kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.928839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.928808    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rfwsn\""
Nov 02 17:26:38.928839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.928833    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 17:26:38.929034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.928833    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 17:26:38.929731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.929713    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"]
Nov 02 17:26:38.970695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.970670    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrx2\" (UniqueName: \"kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.970831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.970728    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.970831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.970763    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.971107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.971090    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:38.971180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:38.971103    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:39.029084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:39.029055    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrx2\" (UniqueName: \"kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:39.140278 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:39.140242    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:26:39.199417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:39.199370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:39.416864 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:39.416779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:39.510876 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:39.510848    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"]
Nov 02 17:26:39.514394 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:39.514364    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815dbad9_2474_4494_bd57_bbf12fb45c59.slice/crio-2b5e8dd6a4f260592185ffcd03e83d222a33e8b8a7df6d88f91af339714545d4 WatchSource:0}: Error finding container 2b5e8dd6a4f260592185ffcd03e83d222a33e8b8a7df6d88f91af339714545d4: Status 404 returned error can't find the container with id 2b5e8dd6a4f260592185ffcd03e83d222a33e8b8a7df6d88f91af339714545d4
Nov 02 17:26:40.213114 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.213081    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerStarted","Data":"2b5e8dd6a4f260592185ffcd03e83d222a33e8b8a7df6d88f91af339714545d4"}
Nov 02 17:26:40.229216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.229182    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"]
Nov 02 17:26:40.233311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.233293    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.279739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.279706    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.279739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.279741    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkp6\" (UniqueName: \"kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.279957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.279769    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.285093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.285076    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ms7vl\""
Nov 02 17:26:40.335578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.335542    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"]
Nov 02 17:26:40.336202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.336182    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"]
Nov 02 17:26:40.339335 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.339314    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.369899 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.369850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:40.380310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380287    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.380457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380334    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.380457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380385    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.380457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380412    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7kc\" (UniqueName: \"kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.380579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380556    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkp6\" (UniqueName: \"kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.380659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380616    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.380732 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380714    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.380773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.380723    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.393318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.393298    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wngx2\""
Nov 02 17:26:40.444580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.444543    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"]
Nov 02 17:26:40.445229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.445206    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"]
Nov 02 17:26:40.448402 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.448381    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"]
Nov 02 17:26:40.448538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.448522    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.449686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.449662    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkp6\" (UniqueName: \"kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.451604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.451587    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.481538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481511    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.481696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481543    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.481696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481566    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7kc\" (UniqueName: \"kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.481696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481583    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhd2\" (UniqueName: \"kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.481696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481687    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.481897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481706    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.481897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481735    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfh9\" (UniqueName: \"kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.481897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481805    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.481897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481884    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.482054 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.481985    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.482120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.482103    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.485021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.484992    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"]
Nov 02 17:26:40.500873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.500846    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-pnmms\""
Nov 02 17:26:40.504763 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.504744    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-rgq8g\""
Nov 02 17:26:40.510601 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.510571    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"]
Nov 02 17:26:40.527111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.527081    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"]
Nov 02 17:26:40.530598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.530580    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.538706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.538683    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7kc\" (UniqueName: \"kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.548618 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.548596    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:26:40.583233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583192    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7cz\" (UniqueName: \"kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.583395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583242    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.583395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583269    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.583395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583379    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfh9\" (UniqueName: \"kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.583540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583417    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.583540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583466    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.583540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583514    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.583703 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583544    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.583703 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583579    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhd2\" (UniqueName: \"kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.583703 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583646    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.583892 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583872    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.583994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.583968    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.584106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.584069    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.589832 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.589801    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"]
Nov 02 17:26:40.607296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.607168    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-c72pw\""
Nov 02 17:26:40.653682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.653651    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:26:40.664858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.664833    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhd2\" (UniqueName: \"kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.683961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.683905    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfh9\" (UniqueName: \"kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.684128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.683995    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.684128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.684049    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.684128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.684117    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7cz\" (UniqueName: \"kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.684419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.684393    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.684545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.684530    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.743398 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.743361    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7cz\" (UniqueName: \"kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.770749 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.770719    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:26:40.777400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.777369    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:26:40.814321 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.814282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:40.851067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.850782    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:26:40.941066 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:40.941039    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"]
Nov 02 17:26:40.941600 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:40.941574    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b9bfb7_ca6a_4350_a61f_970c21b84466.slice/crio-05f058f0e6bcd12e068079af171d0c95615f6409265e88909dabdfb2096e9a27 WatchSource:0}: Error finding container 05f058f0e6bcd12e068079af171d0c95615f6409265e88909dabdfb2096e9a27: Status 404 returned error can't find the container with id 05f058f0e6bcd12e068079af171d0c95615f6409265e88909dabdfb2096e9a27
Nov 02 17:26:41.045874 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.045799    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"]
Nov 02 17:26:41.051756 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:41.051723    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83951b5e_cc98_4e6f_a219_1f5fa01b31dd.slice/crio-d759390cf330becb9d66552c41bded482288da367ef53658260eaa9f8bcaa9a1 WatchSource:0}: Error finding container d759390cf330becb9d66552c41bded482288da367ef53658260eaa9f8bcaa9a1: Status 404 returned error can't find the container with id d759390cf330becb9d66552c41bded482288da367ef53658260eaa9f8bcaa9a1
Nov 02 17:26:41.159042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.159003    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"]
Nov 02 17:26:41.162857 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:41.162826    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fe04e4_b3d4_4e13_a470_8d6f7634fc58.slice/crio-854f1e1cb36ecb096038fbd0c6b88d23c3f6d10ffabd85da4e049aefe393200a WatchSource:0}: Error finding container 854f1e1cb36ecb096038fbd0c6b88d23c3f6d10ffabd85da4e049aefe393200a: Status 404 returned error can't find the container with id 854f1e1cb36ecb096038fbd0c6b88d23c3f6d10ffabd85da4e049aefe393200a
Nov 02 17:26:41.173369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.173344    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"]
Nov 02 17:26:41.176669 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:41.176641    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147470ac_1dc2_4dfd_aa18_0afc2db22fa6.slice/crio-575820b3ee77a0c308b3e80ea4f097e0f5336b22000c6c54f9a7f55f143f6686 WatchSource:0}: Error finding container 575820b3ee77a0c308b3e80ea4f097e0f5336b22000c6c54f9a7f55f143f6686: Status 404 returned error can't find the container with id 575820b3ee77a0c308b3e80ea4f097e0f5336b22000c6c54f9a7f55f143f6686
Nov 02 17:26:41.219123 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.219076    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn" event={"ID":"83951b5e-cc98-4e6f-a219-1f5fa01b31dd","Type":"ContainerStarted","Data":"d759390cf330becb9d66552c41bded482288da367ef53658260eaa9f8bcaa9a1"}
Nov 02 17:26:41.220461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.220417    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" event={"ID":"44b9bfb7-ca6a-4350-a61f-970c21b84466","Type":"ContainerStarted","Data":"05f058f0e6bcd12e068079af171d0c95615f6409265e88909dabdfb2096e9a27"}
Nov 02 17:26:41.221918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.221890    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" event={"ID":"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58","Type":"ContainerStarted","Data":"854f1e1cb36ecb096038fbd0c6b88d23c3f6d10ffabd85da4e049aefe393200a"}
Nov 02 17:26:41.223098 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.223071    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx" event={"ID":"147470ac-1dc2-4dfd-aa18-0afc2db22fa6","Type":"ContainerStarted","Data":"575820b3ee77a0c308b3e80ea4f097e0f5336b22000c6c54f9a7f55f143f6686"}
Nov 02 17:26:41.242150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:41.242116    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"]
Nov 02 17:26:41.246106 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:41.246081    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944f2d06_866e_4a9b_82ee_eeac48f220b5.slice/crio-5134e17d062610d884dbacfae1990c654fe1b0c207dc71554a1a6d3ee8b31bd5 WatchSource:0}: Error finding container 5134e17d062610d884dbacfae1990c654fe1b0c207dc71554a1a6d3ee8b31bd5: Status 404 returned error can't find the container with id 5134e17d062610d884dbacfae1990c654fe1b0c207dc71554a1a6d3ee8b31bd5
Nov 02 17:26:42.231959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.231920    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g" event={"ID":"944f2d06-866e-4a9b-82ee-eeac48f220b5","Type":"ContainerStarted","Data":"5134e17d062610d884dbacfae1990c654fe1b0c207dc71554a1a6d3ee8b31bd5"}
Nov 02 17:26:42.236615 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.236589    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"]
Nov 02 17:26:42.240658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.240621    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.297643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.297589    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"]
Nov 02 17:26:42.297830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.297812    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-9l75l\""
Nov 02 17:26:42.299612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.299353    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.299612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.299392    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkrb\" (UniqueName: \"kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.299612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.299466    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.400732 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.400691    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.400925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.400744    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkrb\" (UniqueName: \"kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.400925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.400818    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.401271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.401247    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.401503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.401485    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.464605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.464562    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkrb\" (UniqueName: \"kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.561524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.561213    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:26:42.966164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:42.966128    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"]
Nov 02 17:26:42.972607 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:42.972576    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4652dd_95d0_4c4d_bf0d_e6eaecba08c0.slice/crio-10463169a0b793c7a1d8f893e6a1582ba996b1c9278435ba0dbace2903a22e8a WatchSource:0}: Error finding container 10463169a0b793c7a1d8f893e6a1582ba996b1c9278435ba0dbace2903a22e8a: Status 404 returned error can't find the container with id 10463169a0b793c7a1d8f893e6a1582ba996b1c9278435ba0dbace2903a22e8a
Nov 02 17:26:43.245978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:43.245778    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n" event={"ID":"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0","Type":"ContainerStarted","Data":"10463169a0b793c7a1d8f893e6a1582ba996b1c9278435ba0dbace2903a22e8a"}
Nov 02 17:26:44.671191 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.671134    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"]
Nov 02 17:26:44.675938 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.675914    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.710571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.710527    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:44.727088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.726989    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.727088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.727059    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.727333 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.727189    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clpn5\" (UniqueName: \"kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.736374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.736284    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"]
Nov 02 17:26:44.827812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.827781    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.828037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.827844    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.828037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.827905    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clpn5\" (UniqueName: \"kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.828654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.828615    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.828912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.828886    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.912496 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.912462    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clpn5\" (UniqueName: \"kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:44.997816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:44.997740    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:26:45.050447 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:45.050311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:45.544673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:45.544617    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"]
Nov 02 17:26:45.554457 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:45.554372    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cd5fae_1e12_419c_a127_3366895815ee.slice/crio-d9850ab9a8432dd94d929312929d1d027ecf04705f139b529bcebfd9b7afcffb WatchSource:0}: Error finding container d9850ab9a8432dd94d929312929d1d027ecf04705f139b529bcebfd9b7afcffb: Status 404 returned error can't find the container with id d9850ab9a8432dd94d929312929d1d027ecf04705f139b529bcebfd9b7afcffb
Nov 02 17:26:45.968789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:45.967133    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"]
Nov 02 17:26:45.976318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:45.972204    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.025589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.024703    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 17:26:46.025589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.025403    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\""
Nov 02 17:26:46.025589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.025497    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 17:26:46.025589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.025557    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 17:26:46.031032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.031001    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"]
Nov 02 17:26:46.044596 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044560    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044609    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044661    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044699    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044724    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbhn\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044752    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044798    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044822    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.044902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.044869    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.146445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146327    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.146445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146389    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146724    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbhn\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146773    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146822    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146848    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146900    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146955    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.147229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.146981    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.148599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.147816    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.148599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.148106    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.148599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.148192    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.148599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.148375    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.149198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.149137    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.160745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.160521    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.174811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.174725    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:46.183202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.183168    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.209740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.209669    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbhn\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.212694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.212410    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8jc9r\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.270667 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.270567    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" event={"ID":"25cd5fae-1e12-419c-a127-3366895815ee","Type":"ContainerStarted","Data":"d9850ab9a8432dd94d929312929d1d027ecf04705f139b529bcebfd9b7afcffb"}
Nov 02 17:26:46.293832 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.291972    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"]
Nov 02 17:26:46.293832 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.293662    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:26:46.296957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.296931    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349114    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349186    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349218    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349251    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4q9m\" (UniqueName: \"kubernetes.io/projected/54b985a5-9c49-4824-9642-5f9f2fa5d66e-kube-api-access-b4q9m\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349285    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349317    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349347    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.349718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.349450    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.350934 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.350842    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-kpkbl\""
Nov 02 17:26:46.355417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.355346    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"]
Nov 02 17:26:46.437493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.437431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.449994    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450137    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450291    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450335    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450413    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450443    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450474    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4q9m\" (UniqueName: \"kubernetes.io/projected/54b985a5-9c49-4824-9642-5f9f2fa5d66e-kube-api-access-b4q9m\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450509    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.451018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.450539    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.453350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.451998    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.453350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.452254    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.453350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.453313    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.454312 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.454252    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.454753 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.454709    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.456126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.456099    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.460694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.460648    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.517240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.517154    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b985a5-9c49-4824-9642-5f9f2fa5d66e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.529705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.529524    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4q9m\" (UniqueName: \"kubernetes.io/projected/54b985a5-9c49-4824-9642-5f9f2fa5d66e-kube-api-access-b4q9m\") pod \"openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc\" (UID: \"54b985a5-9c49-4824-9642-5f9f2fa5d66e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.638950 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.638874    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:46.642241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.641904    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:26:46.721521 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:46.721414    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"]
Nov 02 17:26:47.075182 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.075055    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"]
Nov 02 17:26:47.283355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.283317    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" event={"ID":"96738341-dadc-4422-8c87-4e0a5c634789","Type":"ContainerStarted","Data":"f5182f2d30526fe971a90462cbd630e1469b9bca4728a0c4fce6a2d74f1385bc"}
Nov 02 17:26:47.291462 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.291419    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc" event={"ID":"54b985a5-9c49-4824-9642-5f9f2fa5d66e","Type":"ContainerStarted","Data":"259aa471da6d0d3a42d54a6413e6cf75a8d35f14eb94a19719ef61047c99790e"}
Nov 02 17:26:47.601368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.599115    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"]
Nov 02 17:26:47.604851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.604821    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.630696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.630665    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"]
Nov 02 17:26:47.635564 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.635540    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.660564 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.660451    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"]
Nov 02 17:26:47.663888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.663596    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.663888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.663652    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.663888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.663716    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jxm\" (UniqueName: \"kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.680918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.679311    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"]
Nov 02 17:26:47.683078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.683062    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.693809 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.693783    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"]
Nov 02 17:26:47.698264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.698245    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.710037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.709030    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"]
Nov 02 17:26:47.752296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.752263    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"]
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765084    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jxm\" (UniqueName: \"kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765144    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765175    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzcs\" (UniqueName: \"kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765220    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765254    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765275    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rcj\" (UniqueName: \"kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765292    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765318    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765353    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765384    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765405    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.765424    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgpt\" (UniqueName: \"kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.766845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.766724    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.772479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.772428    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.773762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.773739    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"]
Nov 02 17:26:47.842533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.842502    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jxm\" (UniqueName: \"kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.866185 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866148    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzcs\" (UniqueName: \"kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.866376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866228    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.866376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866277    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rcj\" (UniqueName: \"kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.866376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866305    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.866376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866360    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.866585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866400    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.866585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866429    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.866585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866467    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgpt\" (UniqueName: \"kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.866585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866510    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.869279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.866954    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.869279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.867648    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.869279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.868233    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.869279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.868475    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.869279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.868758    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.869279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.869022    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.930841 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.930807    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:26:47.950240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.939124    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rcj\" (UniqueName: \"kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:47.950240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.940859    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgpt\" (UniqueName: \"kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:47.950240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.941390    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzcs\" (UniqueName: \"kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:47.955462 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:47.955433    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:26:48.018419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:48.018391    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:26:48.030973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:48.030761    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:26:48.338019 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:48.337983    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"]
Nov 02 17:26:48.480214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:48.479946    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"]
Nov 02 17:26:48.480214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:48.480020    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"]
Nov 02 17:26:48.495274 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:26:48.495241    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa0dbcd_aacf_4b94_bd8d_6dbf07a47c73.slice/crio-f4c0da5948b17d70be9bf782e896667b08d2be92e5a0f63b6cced1568b816393 WatchSource:0}: Error finding container f4c0da5948b17d70be9bf782e896667b08d2be92e5a0f63b6cced1568b816393: Status 404 returned error can't find the container with id f4c0da5948b17d70be9bf782e896667b08d2be92e5a0f63b6cced1568b816393
Nov 02 17:26:48.528643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:48.528589    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"]
Nov 02 17:26:49.403161 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:49.403007    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d" event={"ID":"c304518f-84c5-4ec3-ac17-f79233dd87d4","Type":"ContainerStarted","Data":"c405a5b2d5381f997d9d528a0a5539dac58c7a25571042abe85f7121b40d41f8"}
Nov 02 17:26:49.430413 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:49.423598    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" event={"ID":"18cde11c-19ba-444b-b8af-646a219b7fe0","Type":"ContainerStarted","Data":"d5b317e50f66a9afb33c71715a2aa8a6de240ae031096ede37d4eeb0030af77a"}
Nov 02 17:26:49.437855 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:49.437759    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk" event={"ID":"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2","Type":"ContainerStarted","Data":"84d550faed354da2cf6bd6e45408ff62a2d9b6764f2e4689fb421ed93a00f843"}
Nov 02 17:26:49.464619 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:49.464583    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622" event={"ID":"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73","Type":"ContainerStarted","Data":"f4c0da5948b17d70be9bf782e896667b08d2be92e5a0f63b6cced1568b816393"}
Nov 02 17:26:50.122320 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:50.122217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:55.381639 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:26:55.381584    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:26:59.089339 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:26:59.089251    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:27:00.382284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.381231    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:27:00.382284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.381319    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:27:00.382284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.381435    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:27:00.614847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.613893    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" event={"ID":"96738341-dadc-4422-8c87-4e0a5c634789","Type":"ContainerStarted","Data":"bb0721961fc214e4255f946a100603ba98731c86073dc1be72269f296159a7e9"}
Nov 02 17:27:00.696343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.695848    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:27:00.696343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.695936    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:27:00.696343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.695977    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:27:00.783878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:00.782347    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podStartSLOduration=2.145694456 podStartE2EDuration="15.782327861s" totalImagesPullingTime="13.636633405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:00.779405679 +0000 UTC m=+251.620011613" watchObservedRunningTime="2026-11-02 17:27:00.782327861 +0000 UTC m=+251.622933794"
Nov 02 17:27:01.294840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.294811    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:27:01.295124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.295106    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:27:01.307846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.307823    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:27:01.626120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.626079    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc" event={"ID":"54b985a5-9c49-4824-9642-5f9f2fa5d66e","Type":"ContainerStarted","Data":"fee98dfb941cfbb1b54003c0d2add2ad2e0a7522b0351be36609130a506409ec"}
Nov 02 17:27:01.630518 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.630482    4476 generic.go:356] "Generic (PLEG): container finished" podID="25cd5fae-1e12-419c-a127-3366895815ee" containerID="8f385615371a561e5681d0bde2f10e7e5c1d9287c182b0bb121c861492e746ad" exitCode=0
Nov 02 17:27:01.630706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.630645    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" event={"ID":"25cd5fae-1e12-419c-a127-3366895815ee","Type":"ContainerDied","Data":"8f385615371a561e5681d0bde2f10e7e5c1d9287c182b0bb121c861492e746ad"}
Nov 02 17:27:01.637937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.637910    4476 generic.go:356] "Generic (PLEG): container finished" podID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerID="3626c124dd7bd852e1a2d94b07f7b128a765c60c1ce5b7505b85f01832d04eff" exitCode=0
Nov 02 17:27:01.639046 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.638960    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerDied","Data":"3626c124dd7bd852e1a2d94b07f7b128a765c60c1ce5b7505b85f01832d04eff"}
Nov 02 17:27:01.640973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.640834    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:27:01.643504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.643204    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:27:01.643504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.643241    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:27:01.649065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.648963    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:27:01.750511 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:01.750389    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc" podStartSLOduration=2.153597058 podStartE2EDuration="15.750369954s" totalImagesPullingTime="13.596772896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:01.750060711 +0000 UTC m=+252.590666646" watchObservedRunningTime="2026-11-02 17:27:01.750369954 +0000 UTC m=+252.590975895"
Nov 02 17:27:02.648102 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:02.647879    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc"
Nov 02 17:27:03.652318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:03.652290    4476 generic.go:356] "Generic (PLEG): container finished" podID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerID="2670351f38394dce5282f2782792d9968451cc996fd212f07b346a8335e2bc19" exitCode=0
Nov 02 17:27:03.652757 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:03.652372    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk" event={"ID":"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2","Type":"ContainerDied","Data":"2670351f38394dce5282f2782792d9968451cc996fd212f07b346a8335e2bc19"}
Nov 02 17:27:04.664097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.664016    4476 generic.go:356] "Generic (PLEG): container finished" podID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerID="89c1cbd8ee36e34f5098cfe3e22a10353eb181328727d4992e44de910218c725" exitCode=0
Nov 02 17:27:04.664097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.664071    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx" event={"ID":"147470ac-1dc2-4dfd-aa18-0afc2db22fa6","Type":"ContainerDied","Data":"89c1cbd8ee36e34f5098cfe3e22a10353eb181328727d4992e44de910218c725"}
Nov 02 17:27:04.666188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.666164    4476 generic.go:356] "Generic (PLEG): container finished" podID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerID="6886c8a3c0b8d74c975ee148eac20c35f27a3ca33746688404a80126a633f461" exitCode=0
Nov 02 17:27:04.666309 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.666247    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g" event={"ID":"944f2d06-866e-4a9b-82ee-eeac48f220b5","Type":"ContainerDied","Data":"6886c8a3c0b8d74c975ee148eac20c35f27a3ca33746688404a80126a633f461"}
Nov 02 17:27:04.668157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.668130    4476 generic.go:356] "Generic (PLEG): container finished" podID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerID="f4be1b66ee490f16565dc4aa5ac38ab787239358c3df7a4fe4ed322b5d4a0e52" exitCode=0
Nov 02 17:27:04.668261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.668219    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d" event={"ID":"c304518f-84c5-4ec3-ac17-f79233dd87d4","Type":"ContainerDied","Data":"f4be1b66ee490f16565dc4aa5ac38ab787239358c3df7a4fe4ed322b5d4a0e52"}
Nov 02 17:27:04.670145 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.670123    4476 generic.go:356] "Generic (PLEG): container finished" podID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerID="80e3fdcf4a8b1694f88b94232c627eedc8646286293500913501d2f4b154f08e" exitCode=0
Nov 02 17:27:04.670415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.670192    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" event={"ID":"18cde11c-19ba-444b-b8af-646a219b7fe0","Type":"ContainerDied","Data":"80e3fdcf4a8b1694f88b94232c627eedc8646286293500913501d2f4b154f08e"}
Nov 02 17:27:04.672201 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.672130    4476 generic.go:356] "Generic (PLEG): container finished" podID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerID="06cdc66ef1ecb66af9f00243dcdebe6d6747c66298697dd49dc676d6b7bcd017" exitCode=0
Nov 02 17:27:04.672283 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.672204    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n" event={"ID":"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0","Type":"ContainerDied","Data":"06cdc66ef1ecb66af9f00243dcdebe6d6747c66298697dd49dc676d6b7bcd017"}
Nov 02 17:27:04.674514 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.674489    4476 generic.go:356] "Generic (PLEG): container finished" podID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerID="40188be19c1d7a18a539d21d709421c2f316602e775508e4b8d7060f2d030e5d" exitCode=0
Nov 02 17:27:04.674621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.674567    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn" event={"ID":"83951b5e-cc98-4e6f-a219-1f5fa01b31dd","Type":"ContainerDied","Data":"40188be19c1d7a18a539d21d709421c2f316602e775508e4b8d7060f2d030e5d"}
Nov 02 17:27:04.676691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.676667    4476 generic.go:356] "Generic (PLEG): container finished" podID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerID="c6460cc868ba43a2800ecd6a8ef3c4208521432307171d2e66efe347221b9fd0" exitCode=0
Nov 02 17:27:04.676801 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.676695    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" event={"ID":"44b9bfb7-ca6a-4350-a61f-970c21b84466","Type":"ContainerDied","Data":"c6460cc868ba43a2800ecd6a8ef3c4208521432307171d2e66efe347221b9fd0"}
Nov 02 17:27:04.678570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.678551    4476 generic.go:356] "Generic (PLEG): container finished" podID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerID="b08c8c0ff79185282362ed4a1ff189febbdb1a5c34e8bc599d4676c167f3ee6e" exitCode=0
Nov 02 17:27:04.678684 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.678602    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" event={"ID":"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58","Type":"ContainerDied","Data":"b08c8c0ff79185282362ed4a1ff189febbdb1a5c34e8bc599d4676c167f3ee6e"}
Nov 02 17:27:04.680737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.680718    4476 generic.go:356] "Generic (PLEG): container finished" podID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerID="ffe4366f79778a95268b10b929d337f34f338d52a78f1cd867d03d4fa87742e0" exitCode=0
Nov 02 17:27:04.680895 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:04.680875    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622" event={"ID":"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73","Type":"ContainerDied","Data":"ffe4366f79778a95268b10b929d337f34f338d52a78f1cd867d03d4fa87742e0"}
Nov 02 17:27:05.688164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:05.688135    4476 generic.go:356] "Generic (PLEG): container finished" podID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerID="ecb9ad0f7cc270d4cce3f25a1bb8ce8efbe664ce22082c44b51e41d43e0bffa9" exitCode=0
Nov 02 17:27:05.688576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:05.688205    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk" event={"ID":"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2","Type":"ContainerDied","Data":"ecb9ad0f7cc270d4cce3f25a1bb8ce8efbe664ce22082c44b51e41d43e0bffa9"}
Nov 02 17:27:05.690395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:05.690373    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerStarted","Data":"8f4c0929c64f1fd1a8c62800fa8b64553ea93c2f35a603d61821ae638559fe66"}
Nov 02 17:27:06.699748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.699714    4476 generic.go:356] "Generic (PLEG): container finished" podID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerID="358f1d13417b74e366dbaf7916f356a1efe12ed3cc5a25fc2f7c1c0ba43ea796" exitCode=0
Nov 02 17:27:06.700151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.699795    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" event={"ID":"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58","Type":"ContainerDied","Data":"358f1d13417b74e366dbaf7916f356a1efe12ed3cc5a25fc2f7c1c0ba43ea796"}
Nov 02 17:27:06.704214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.703598    4476 generic.go:356] "Generic (PLEG): container finished" podID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerID="4aae4c3d466473a89fbbb64771be515ac709607e00ae2ac51d7aa64c9c44238f" exitCode=0
Nov 02 17:27:06.704214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.703837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx" event={"ID":"147470ac-1dc2-4dfd-aa18-0afc2db22fa6","Type":"ContainerDied","Data":"4aae4c3d466473a89fbbb64771be515ac709607e00ae2ac51d7aa64c9c44238f"}
Nov 02 17:27:06.708789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.708765    4476 generic.go:356] "Generic (PLEG): container finished" podID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerID="8f4c0929c64f1fd1a8c62800fa8b64553ea93c2f35a603d61821ae638559fe66" exitCode=0
Nov 02 17:27:06.708886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.708874    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerDied","Data":"8f4c0929c64f1fd1a8c62800fa8b64553ea93c2f35a603d61821ae638559fe66"}
Nov 02 17:27:06.712929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:06.712907    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn" event={"ID":"83951b5e-cc98-4e6f-a219-1f5fa01b31dd","Type":"ContainerStarted","Data":"614fe99fe5b373f4d53dc1c791d63fe544c02e689db8598f3af06109cc130dd5"}
Nov 02 17:27:07.726347 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.725785    4476 generic.go:356] "Generic (PLEG): container finished" podID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerID="1a5d7cb57fb835dd8a7b31c7484d81bc501425f8bfbaa8f10d15956d795e68bb" exitCode=0
Nov 02 17:27:07.726347 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.725902    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g" event={"ID":"944f2d06-866e-4a9b-82ee-eeac48f220b5","Type":"ContainerDied","Data":"1a5d7cb57fb835dd8a7b31c7484d81bc501425f8bfbaa8f10d15956d795e68bb"}
Nov 02 17:27:07.733088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.733056    4476 generic.go:356] "Generic (PLEG): container finished" podID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerID="70adee001c711428b76722f1fa6872389b9686cea5273994f01fa170d31c1a0f" exitCode=0
Nov 02 17:27:07.733227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.733207    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d" event={"ID":"c304518f-84c5-4ec3-ac17-f79233dd87d4","Type":"ContainerDied","Data":"70adee001c711428b76722f1fa6872389b9686cea5273994f01fa170d31c1a0f"}
Nov 02 17:27:07.738785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.738755    4476 generic.go:356] "Generic (PLEG): container finished" podID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerID="614fe99fe5b373f4d53dc1c791d63fe544c02e689db8598f3af06109cc130dd5" exitCode=0
Nov 02 17:27:07.738883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.738852    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn" event={"ID":"83951b5e-cc98-4e6f-a219-1f5fa01b31dd","Type":"ContainerDied","Data":"614fe99fe5b373f4d53dc1c791d63fe544c02e689db8598f3af06109cc130dd5"}
Nov 02 17:27:07.742507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.742441    4476 generic.go:356] "Generic (PLEG): container finished" podID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerID="c95a03994d2a4e4c9da5175e80e24002fc008fa60458b06f698e070e1baca2fb" exitCode=0
Nov 02 17:27:07.742707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.742612    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" event={"ID":"44b9bfb7-ca6a-4350-a61f-970c21b84466","Type":"ContainerDied","Data":"c95a03994d2a4e4c9da5175e80e24002fc008fa60458b06f698e070e1baca2fb"}
Nov 02 17:27:07.746612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.746578    4476 generic.go:356] "Generic (PLEG): container finished" podID="25cd5fae-1e12-419c-a127-3366895815ee" containerID="53063a606529214129cd65c9cba7ecf5ba122e5c2868d5ca01f41c39e95f4883" exitCode=0
Nov 02 17:27:07.746759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:07.746649    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" event={"ID":"25cd5fae-1e12-419c-a127-3366895815ee","Type":"ContainerDied","Data":"53063a606529214129cd65c9cba7ecf5ba122e5c2868d5ca01f41c39e95f4883"}
Nov 02 17:27:09.199777 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:09.199659    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:09.399265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:09.399197    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:12.985794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:10.785847    4476 generic.go:356] "Generic (PLEG): container finished" podID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerID="11f4b8929c94901ebce02c3edf3800cd1465b5179d7f5f32d8f7625558a834e4" exitCode=0
Nov 02 17:27:12.985794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:10.785919    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n" event={"ID":"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0","Type":"ContainerDied","Data":"11f4b8929c94901ebce02c3edf3800cd1465b5179d7f5f32d8f7625558a834e4"}
Nov 02 17:27:16.902509 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:16.902457    4476 generic.go:356] "Generic (PLEG): container finished" podID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerID="984e2bb9aed61fb569527660b0324841ace92537469d77c901ae3ee368f440fc" exitCode=0
Nov 02 17:27:16.903044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:16.902617    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622" event={"ID":"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73","Type":"ContainerDied","Data":"984e2bb9aed61fb569527660b0324841ace92537469d77c901ae3ee368f440fc"}
Nov 02 17:27:16.926883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:16.925948    4476 generic.go:356] "Generic (PLEG): container finished" podID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerID="972b60d03caa92a11836670e44c48a3a81398cc13629ab1c5b882e64b53c9569" exitCode=0
Nov 02 17:27:16.926883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:16.926044    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" event={"ID":"18cde11c-19ba-444b-b8af-646a219b7fe0","Type":"ContainerDied","Data":"972b60d03caa92a11836670e44c48a3a81398cc13629ab1c5b882e64b53c9569"}
Nov 02 17:27:28.071319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:28.071257    4476 generic.go:356] "Generic (PLEG): container finished" podID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerID="70b0b586df6980eac6ce6034a4091f8c72032c982d42b7751cf90bd7aeaa6497" exitCode=0
Nov 02 17:27:28.071873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:28.071399    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk" event={"ID":"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2","Type":"ContainerDied","Data":"70b0b586df6980eac6ce6034a4091f8c72032c982d42b7751cf90bd7aeaa6497"}
Nov 02 17:27:29.087718 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:27:29.087680    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:27:29.273297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.272969    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:27:29.438554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.438165    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle\") pod \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") "
Nov 02 17:27:29.438554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.438257    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util\") pod \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") "
Nov 02 17:27:29.438554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.438304    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt\" (UniqueName: \"kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt\") pod \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\" (UID: \"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2\") "
Nov 02 17:27:29.440189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.440132    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle" pod "cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" (UID: "cee7fcc2-0d4d-466c-9d91-2d11144d2bb2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:29.440396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.440348    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:29.448023 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.447994    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util" pod "cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" (UID: "cee7fcc2-0d4d-466c-9d91-2d11144d2bb2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:29.451526 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.451482    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt" pod "cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" (UID: "cee7fcc2-0d4d-466c-9d91-2d11144d2bb2"). InnerVolumeSpecName "kube-api-access-pdgpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:29.541143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.541043    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:29.541143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:29.541074    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdgpt\" (UniqueName: \"kubernetes.io/projected/cee7fcc2-0d4d-466c-9d91-2d11144d2bb2-kube-api-access-pdgpt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:30.094350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.094323    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" event={"ID":"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58","Type":"ContainerStarted","Data":"3a5c2fcdb3ec8dde758c127afad90918dff882dd110c4714b435868744cfca63"}
Nov 02 17:27:30.101000 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.100978    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk"
Nov 02 17:27:30.101111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.101022    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fw5bk" event={"ID":"cee7fcc2-0d4d-466c-9d91-2d11144d2bb2","Type":"ContainerDied","Data":"84d550faed354da2cf6bd6e45408ff62a2d9b6764f2e4689fb421ed93a00f843"}
Nov 02 17:27:30.101111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.101061    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84d550faed354da2cf6bd6e45408ff62a2d9b6764f2e4689fb421ed93a00f843"
Nov 02 17:27:30.112663 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.112063    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerStarted","Data":"c1550da400f5f1704055c9e968fda2f82706cc6214ce48bce56955d13dc91b2e"}
Nov 02 17:27:30.214079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.214011    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" podStartSLOduration=3.319369471 podStartE2EDuration="50.213994369s" totalImagesPullingTime="46.894624898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:30.213167612 +0000 UTC m=+281.053773551" watchObservedRunningTime="2026-11-02 17:27:30.213994369 +0000 UTC m=+281.054600305"
Nov 02 17:27:30.333196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:30.329412    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" podStartSLOduration=4.693043627 podStartE2EDuration="52.329398245s" totalImagesPullingTime="47.636354618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:30.328921338 +0000 UTC m=+281.169527273" watchObservedRunningTime="2026-11-02 17:27:30.329398245 +0000 UTC m=+281.170004179"
Nov 02 17:27:31.136177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.135545    4476 generic.go:356] "Generic (PLEG): container finished" podID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerID="c3ecc4f0db8321f6e2efb046a29d27f635a7fc88145d593f5e247767cc722dfe" exitCode=0
Nov 02 17:27:31.136177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.135699    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx" event={"ID":"147470ac-1dc2-4dfd-aa18-0afc2db22fa6","Type":"ContainerDied","Data":"c3ecc4f0db8321f6e2efb046a29d27f635a7fc88145d593f5e247767cc722dfe"}
Nov 02 17:27:31.141281 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.141252    4476 generic.go:356] "Generic (PLEG): container finished" podID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerID="c1550da400f5f1704055c9e968fda2f82706cc6214ce48bce56955d13dc91b2e" exitCode=0
Nov 02 17:27:31.141596 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.141449    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerDied","Data":"c1550da400f5f1704055c9e968fda2f82706cc6214ce48bce56955d13dc91b2e"}
Nov 02 17:27:31.147339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.147310    4476 generic.go:356] "Generic (PLEG): container finished" podID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerID="1729f2dad9b4fbff56f359e053e3f35ac832893547b22f2f89d00f2a821163a2" exitCode=0
Nov 02 17:27:31.147452 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.147392    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn" event={"ID":"83951b5e-cc98-4e6f-a219-1f5fa01b31dd","Type":"ContainerDied","Data":"1729f2dad9b4fbff56f359e053e3f35ac832893547b22f2f89d00f2a821163a2"}
Nov 02 17:27:31.150571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.150546    4476 generic.go:356] "Generic (PLEG): container finished" podID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerID="3a5c2fcdb3ec8dde758c127afad90918dff882dd110c4714b435868744cfca63" exitCode=0
Nov 02 17:27:31.150694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:31.150584    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" event={"ID":"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58","Type":"ContainerDied","Data":"3a5c2fcdb3ec8dde758c127afad90918dff882dd110c4714b435868744cfca63"}
Nov 02 17:27:32.158847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.158818    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" event={"ID":"25cd5fae-1e12-419c-a127-3366895815ee","Type":"ContainerStarted","Data":"d9a242f1f5d38f29eb99304a88332e9558a9651f3e48a33a88960381716c492b"}
Nov 02 17:27:32.161565 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.161537    4476 generic.go:356] "Generic (PLEG): container finished" podID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerID="a098a3004af4527ad88e1786301c29f4311a1b62a1096af3f8837d16b1a0c0b1" exitCode=0
Nov 02 17:27:32.161818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.161799    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622" event={"ID":"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73","Type":"ContainerDied","Data":"a098a3004af4527ad88e1786301c29f4311a1b62a1096af3f8837d16b1a0c0b1"}
Nov 02 17:27:32.165903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.165842    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" event={"ID":"18cde11c-19ba-444b-b8af-646a219b7fe0","Type":"ContainerStarted","Data":"45402f3a0fc7159b0cbd027d5866ef01bcc5a6a2f647d3f27f41fe84d78f4f69"}
Nov 02 17:27:32.278084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.277750    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" podStartSLOduration=2.870324671 podStartE2EDuration="48.277730626s" totalImagesPullingTime="45.407405955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:32.274793523 +0000 UTC m=+283.115399458" watchObservedRunningTime="2026-11-02 17:27:32.277730626 +0000 UTC m=+283.118336561"
Nov 02 17:27:32.406373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.406337    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:27:32.443515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.443003    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:27:32.483912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.483883    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:27:32.496396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.496366    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:27:32.505350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.505301    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" podStartSLOduration=4.149655093 podStartE2EDuration="45.50528642s" totalImagesPullingTime="41.355631327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:32.504685514 +0000 UTC m=+283.345291447" watchObservedRunningTime="2026-11-02 17:27:32.50528642 +0000 UTC m=+283.345892353"
Nov 02 17:27:32.572620 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.572594    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9\" (UniqueName: \"kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9\") pod \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") "
Nov 02 17:27:32.572786 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.572659    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util\") pod \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") "
Nov 02 17:27:32.572852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.572784    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2\" (UniqueName: \"kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2\") pod \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") "
Nov 02 17:27:32.572906 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.572870    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util\") pod \"815dbad9-2474-4494-bd57-bbf12fb45c59\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") "
Nov 02 17:27:32.572963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.572934    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util\") pod \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") "
Nov 02 17:27:32.573020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.572973    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle\") pod \"815dbad9-2474-4494-bd57-bbf12fb45c59\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") "
Nov 02 17:27:32.573077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.573020    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle\") pod \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\" (UID: \"147470ac-1dc2-4dfd-aa18-0afc2db22fa6\") "
Nov 02 17:27:32.573077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.573051    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle\") pod \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\" (UID: \"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58\") "
Nov 02 17:27:32.573199 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.573097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2\" (UniqueName: \"kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2\") pod \"815dbad9-2474-4494-bd57-bbf12fb45c59\" (UID: \"815dbad9-2474-4494-bd57-bbf12fb45c59\") "
Nov 02 17:27:32.574010 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.573978    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle" pod "147470ac-1dc2-4dfd-aa18-0afc2db22fa6" (UID: "147470ac-1dc2-4dfd-aa18-0afc2db22fa6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.574709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.574680    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle" pod "c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" (UID: "c3fe04e4-b3d4-4e13-a470-8d6f7634fc58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.575147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.575122    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9" pod "c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" (UID: "c3fe04e4-b3d4-4e13-a470-8d6f7634fc58"). InnerVolumeSpecName "kube-api-access-tmfh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:32.575372 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.575352    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle" pod "815dbad9-2474-4494-bd57-bbf12fb45c59" (UID: "815dbad9-2474-4494-bd57-bbf12fb45c59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.575660 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.575616    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2" pod "147470ac-1dc2-4dfd-aa18-0afc2db22fa6" (UID: "147470ac-1dc2-4dfd-aa18-0afc2db22fa6"). InnerVolumeSpecName "kube-api-access-jrhd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:32.575928 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.575907    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2" pod "815dbad9-2474-4494-bd57-bbf12fb45c59" (UID: "815dbad9-2474-4494-bd57-bbf12fb45c59"). InnerVolumeSpecName "kube-api-access-rvrx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:32.580779 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.580755    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util" pod "147470ac-1dc2-4dfd-aa18-0afc2db22fa6" (UID: "147470ac-1dc2-4dfd-aa18-0afc2db22fa6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.580948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.580926    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util" pod "815dbad9-2474-4494-bd57-bbf12fb45c59" (UID: "815dbad9-2474-4494-bd57-bbf12fb45c59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.581760 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.581739    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util" pod "c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" (UID: "c3fe04e4-b3d4-4e13-a470-8d6f7634fc58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.674813 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.674779    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle\") pod \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") "
Nov 02 17:27:32.674985 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.674845    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util\") pod \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") "
Nov 02 17:27:32.674985 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.674876    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc\" (UniqueName: \"kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc\") pod \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\" (UID: \"83951b5e-cc98-4e6f-a219-1f5fa01b31dd\") "
Nov 02 17:27:32.675139 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675123    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675144    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675158    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/815dbad9-2474-4494-bd57-bbf12fb45c59-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675173    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675188    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675202    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrx2\" (UniqueName: \"kubernetes.io/projected/815dbad9-2474-4494-bd57-bbf12fb45c59-kube-api-access-rvrx2\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675216    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmfh9\" (UniqueName: \"kubernetes.io/projected/c3fe04e4-b3d4-4e13-a470-8d6f7634fc58-kube-api-access-tmfh9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675232    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675248    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhd2\" (UniqueName: \"kubernetes.io/projected/147470ac-1dc2-4dfd-aa18-0afc2db22fa6-kube-api-access-jrhd2\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.675381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.675328    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle" pod "83951b5e-cc98-4e6f-a219-1f5fa01b31dd" (UID: "83951b5e-cc98-4e6f-a219-1f5fa01b31dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.677181 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.677160    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc" pod "83951b5e-cc98-4e6f-a219-1f5fa01b31dd" (UID: "83951b5e-cc98-4e6f-a219-1f5fa01b31dd"). InnerVolumeSpecName "kube-api-access-ns7kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:32.680616 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.680592    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util" pod "83951b5e-cc98-4e6f-a219-1f5fa01b31dd" (UID: "83951b5e-cc98-4e6f-a219-1f5fa01b31dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:32.775991 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.775909    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.775991 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.775936    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:32.775991 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:32.775948    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns7kc\" (UniqueName: \"kubernetes.io/projected/83951b5e-cc98-4e6f-a219-1f5fa01b31dd-kube-api-access-ns7kc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:33.171841 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.171804    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn"
Nov 02 17:27:33.172263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.171803    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqkxkn" event={"ID":"83951b5e-cc98-4e6f-a219-1f5fa01b31dd","Type":"ContainerDied","Data":"d759390cf330becb9d66552c41bded482288da367ef53658260eaa9f8bcaa9a1"}
Nov 02 17:27:33.172263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.171925    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d759390cf330becb9d66552c41bded482288da367ef53658260eaa9f8bcaa9a1"
Nov 02 17:27:33.174247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.174175    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" event={"ID":"44b9bfb7-ca6a-4350-a61f-970c21b84466","Type":"ContainerStarted","Data":"3e30933c65bc0355ed8559cb28083d769586fcbf1f73f4b31e91a1a2adbdad9c"}
Nov 02 17:27:33.176773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.176750    4476 generic.go:356] "Generic (PLEG): container finished" podID="25cd5fae-1e12-419c-a127-3366895815ee" containerID="d9a242f1f5d38f29eb99304a88332e9558a9651f3e48a33a88960381716c492b" exitCode=0
Nov 02 17:27:33.176897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.176860    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" event={"ID":"25cd5fae-1e12-419c-a127-3366895815ee","Type":"ContainerDied","Data":"d9a242f1f5d38f29eb99304a88332e9558a9651f3e48a33a88960381716c492b"}
Nov 02 17:27:33.186001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.185904    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4"
Nov 02 17:27:33.186001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.185900    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sfmj4" event={"ID":"c3fe04e4-b3d4-4e13-a470-8d6f7634fc58","Type":"ContainerDied","Data":"854f1e1cb36ecb096038fbd0c6b88d23c3f6d10ffabd85da4e049aefe393200a"}
Nov 02 17:27:33.186117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.186061    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="854f1e1cb36ecb096038fbd0c6b88d23c3f6d10ffabd85da4e049aefe393200a"
Nov 02 17:27:33.188473 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.188447    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx" event={"ID":"147470ac-1dc2-4dfd-aa18-0afc2db22fa6","Type":"ContainerDied","Data":"575820b3ee77a0c308b3e80ea4f097e0f5336b22000c6c54f9a7f55f143f6686"}
Nov 02 17:27:33.188578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.188487    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575820b3ee77a0c308b3e80ea4f097e0f5336b22000c6c54f9a7f55f143f6686"
Nov 02 17:27:33.188578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.188494    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pc9hx"
Nov 02 17:27:33.190685 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.190659    4476 generic.go:356] "Generic (PLEG): container finished" podID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerID="5f6e2b9bc6728f302199b5d8cef715fd8b94659e3841364afc94d2b7b03ed8eb" exitCode=0
Nov 02 17:27:33.190798 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.190685    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g" event={"ID":"944f2d06-866e-4a9b-82ee-eeac48f220b5","Type":"ContainerDied","Data":"5f6e2b9bc6728f302199b5d8cef715fd8b94659e3841364afc94d2b7b03ed8eb"}
Nov 02 17:27:33.192245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.192228    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:27:33.192868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.192844    4476 generic.go:356] "Generic (PLEG): container finished" podID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerID="ce4a9e6476c981292c81f185ffc024f47e5ea235bd68d211ecab1d3464256363" exitCode=1
Nov 02 17:27:33.192967 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.192923    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d" event={"ID":"c304518f-84c5-4ec3-ac17-f79233dd87d4","Type":"ContainerDied","Data":"ce4a9e6476c981292c81f185ffc024f47e5ea235bd68d211ecab1d3464256363"}
Nov 02 17:27:33.195190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.195173    4476 generic.go:356] "Generic (PLEG): container finished" podID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerID="45402f3a0fc7159b0cbd027d5866ef01bcc5a6a2f647d3f27f41fe84d78f4f69" exitCode=0
Nov 02 17:27:33.195335 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.195320    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" event={"ID":"18cde11c-19ba-444b-b8af-646a219b7fe0","Type":"ContainerDied","Data":"45402f3a0fc7159b0cbd027d5866ef01bcc5a6a2f647d3f27f41fe84d78f4f69"}
Nov 02 17:27:33.197550 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.197530    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w"
Nov 02 17:27:33.197682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.197551    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjch6w" event={"ID":"815dbad9-2474-4494-bd57-bbf12fb45c59","Type":"ContainerDied","Data":"2b5e8dd6a4f260592185ffcd03e83d222a33e8b8a7df6d88f91af339714545d4"}
Nov 02 17:27:33.197682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.197590    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b5e8dd6a4f260592185ffcd03e83d222a33e8b8a7df6d88f91af339714545d4"
Nov 02 17:27:33.291683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.291336    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" podStartSLOduration=3.201504161 podStartE2EDuration="53.291320555s" totalImagesPullingTime="50.089816394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:26:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:33.290469428 +0000 UTC m=+284.131075370" watchObservedRunningTime="2026-11-02 17:27:33.291320555 +0000 UTC m=+284.131926489"
Nov 02 17:27:33.322975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.322951    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:27:33.483211 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.483137    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util\") pod \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") "
Nov 02 17:27:33.483211 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.483185    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle\") pod \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") "
Nov 02 17:27:33.483438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.483220    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs\" (UniqueName: \"kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs\") pod \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\" (UID: \"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73\") "
Nov 02 17:27:33.483762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.483737    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle" pod "faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" (UID: "faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:33.485395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.485370    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs" pod "faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" (UID: "faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73"). InnerVolumeSpecName "kube-api-access-ltzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:33.488519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.488498    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util" pod "faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" (UID: "faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:33.584310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.584275    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:33.584310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.584300    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:33.584310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:33.584310    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzcs\" (UniqueName: \"kubernetes.io/projected/faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73-kube-api-access-ltzcs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.203798 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.203765    4476 generic.go:356] "Generic (PLEG): container finished" podID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerID="89830b22099802c6a5c851658da13fe305d24aba387f53d1ccaf0b5f7ec3e4d5" exitCode=0
Nov 02 17:27:34.204192 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.203845    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n" event={"ID":"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0","Type":"ContainerDied","Data":"89830b22099802c6a5c851658da13fe305d24aba387f53d1ccaf0b5f7ec3e4d5"}
Nov 02 17:27:34.205877 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.205852    4476 generic.go:356] "Generic (PLEG): container finished" podID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerID="3e30933c65bc0355ed8559cb28083d769586fcbf1f73f4b31e91a1a2adbdad9c" exitCode=0
Nov 02 17:27:34.206020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.205926    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" event={"ID":"44b9bfb7-ca6a-4350-a61f-970c21b84466","Type":"ContainerDied","Data":"3e30933c65bc0355ed8559cb28083d769586fcbf1f73f4b31e91a1a2adbdad9c"}
Nov 02 17:27:34.207750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.207734    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622"
Nov 02 17:27:34.207869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.207774    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cql622" event={"ID":"faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73","Type":"ContainerDied","Data":"f4c0da5948b17d70be9bf782e896667b08d2be92e5a0f63b6cced1568b816393"}
Nov 02 17:27:34.207869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.207807    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c0da5948b17d70be9bf782e896667b08d2be92e5a0f63b6cced1568b816393"
Nov 02 17:27:34.351345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.351322    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:27:34.382682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.382660    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:27:34.392458 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.392439    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:27:34.394424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.394406    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:27:34.397114 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.397096    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:27:34.491888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.491800    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz\" (UniqueName: \"kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz\") pod \"944f2d06-866e-4a9b-82ee-eeac48f220b5\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") "
Nov 02 17:27:34.491888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.491840    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj\" (UniqueName: \"kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj\") pod \"c304518f-84c5-4ec3-ac17-f79233dd87d4\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") "
Nov 02 17:27:34.491888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.491867    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util\") pod \"25cd5fae-1e12-419c-a127-3366895815ee\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.491904    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5\" (UniqueName: \"kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5\") pod \"25cd5fae-1e12-419c-a127-3366895815ee\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.491938    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle\") pod \"944f2d06-866e-4a9b-82ee-eeac48f220b5\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.491979    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle\") pod \"c304518f-84c5-4ec3-ac17-f79233dd87d4\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492019    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm\" (UniqueName: \"kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm\") pod \"18cde11c-19ba-444b-b8af-646a219b7fe0\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492053    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util\") pod \"c304518f-84c5-4ec3-ac17-f79233dd87d4\" (UID: \"c304518f-84c5-4ec3-ac17-f79233dd87d4\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492109    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util\") pod \"18cde11c-19ba-444b-b8af-646a219b7fe0\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") "
Nov 02 17:27:34.492180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492145    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle\") pod \"18cde11c-19ba-444b-b8af-646a219b7fe0\" (UID: \"18cde11c-19ba-444b-b8af-646a219b7fe0\") "
Nov 02 17:27:34.492799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492184    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle\") pod \"25cd5fae-1e12-419c-a127-3366895815ee\" (UID: \"25cd5fae-1e12-419c-a127-3366895815ee\") "
Nov 02 17:27:34.492799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492252    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util\") pod \"944f2d06-866e-4a9b-82ee-eeac48f220b5\" (UID: \"944f2d06-866e-4a9b-82ee-eeac48f220b5\") "
Nov 02 17:27:34.492799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492412    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle" pod "944f2d06-866e-4a9b-82ee-eeac48f220b5" (UID: "944f2d06-866e-4a9b-82ee-eeac48f220b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.493118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.492924    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.493554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.493506    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle" pod "18cde11c-19ba-444b-b8af-646a219b7fe0" (UID: "18cde11c-19ba-444b-b8af-646a219b7fe0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.494212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.494184    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle" pod "25cd5fae-1e12-419c-a127-3366895815ee" (UID: "25cd5fae-1e12-419c-a127-3366895815ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.494409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.494379    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle" pod "c304518f-84c5-4ec3-ac17-f79233dd87d4" (UID: "c304518f-84c5-4ec3-ac17-f79233dd87d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.495118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.495082    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5" pod "25cd5fae-1e12-419c-a127-3366895815ee" (UID: "25cd5fae-1e12-419c-a127-3366895815ee"). InnerVolumeSpecName "kube-api-access-clpn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:34.495984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.495940    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz" pod "944f2d06-866e-4a9b-82ee-eeac48f220b5" (UID: "944f2d06-866e-4a9b-82ee-eeac48f220b5"). InnerVolumeSpecName "kube-api-access-wl7cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:34.496137 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.496113    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj" pod "c304518f-84c5-4ec3-ac17-f79233dd87d4" (UID: "c304518f-84c5-4ec3-ac17-f79233dd87d4"). InnerVolumeSpecName "kube-api-access-h4rcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:34.497950 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.497922    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm" pod "18cde11c-19ba-444b-b8af-646a219b7fe0" (UID: "18cde11c-19ba-444b-b8af-646a219b7fe0"). InnerVolumeSpecName "kube-api-access-24jxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:34.499905 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.499774    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util" pod "25cd5fae-1e12-419c-a127-3366895815ee" (UID: "25cd5fae-1e12-419c-a127-3366895815ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.502069 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.502046    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util" pod "18cde11c-19ba-444b-b8af-646a219b7fe0" (UID: "18cde11c-19ba-444b-b8af-646a219b7fe0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.502387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.502366    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util" pod "c304518f-84c5-4ec3-ac17-f79233dd87d4" (UID: "c304518f-84c5-4ec3-ac17-f79233dd87d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.502915 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.502895    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util" pod "944f2d06-866e-4a9b-82ee-eeac48f220b5" (UID: "944f2d06-866e-4a9b-82ee-eeac48f220b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:34.593524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593490    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593520    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593535    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18cde11c-19ba-444b-b8af-646a219b7fe0-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593549    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593559    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/944f2d06-866e-4a9b-82ee-eeac48f220b5-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593571    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7cz\" (UniqueName: \"kubernetes.io/projected/944f2d06-866e-4a9b-82ee-eeac48f220b5-kube-api-access-wl7cz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593583    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rcj\" (UniqueName: \"kubernetes.io/projected/c304518f-84c5-4ec3-ac17-f79233dd87d4-kube-api-access-h4rcj\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593599    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25cd5fae-1e12-419c-a127-3366895815ee-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593610    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clpn5\" (UniqueName: \"kubernetes.io/projected/25cd5fae-1e12-419c-a127-3366895815ee-kube-api-access-clpn5\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593622    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c304518f-84c5-4ec3-ac17-f79233dd87d4-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:34.593773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:34.593659    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jxm\" (UniqueName: \"kubernetes.io/projected/18cde11c-19ba-444b-b8af-646a219b7fe0-kube-api-access-24jxm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:35.213238 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.213210    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:27:35.213998 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.213980    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d"
Nov 02 17:27:35.214108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.213986    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d" event={"ID":"c304518f-84c5-4ec3-ac17-f79233dd87d4","Type":"ContainerDied","Data":"c405a5b2d5381f997d9d528a0a5539dac58c7a25571042abe85f7121b40d41f8"}
Nov 02 17:27:35.214108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.214041    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c405a5b2d5381f997d9d528a0a5539dac58c7a25571042abe85f7121b40d41f8"
Nov 02 17:27:35.216008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.215984    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k"
Nov 02 17:27:35.216008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.216000    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am987k" event={"ID":"18cde11c-19ba-444b-b8af-646a219b7fe0","Type":"ContainerDied","Data":"d5b317e50f66a9afb33c71715a2aa8a6de240ae031096ede37d4eeb0030af77a"}
Nov 02 17:27:35.216190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.216035    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b317e50f66a9afb33c71715a2aa8a6de240ae031096ede37d4eeb0030af77a"
Nov 02 17:27:35.217833 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.217814    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9"
Nov 02 17:27:35.217931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.217837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cklpl9" event={"ID":"25cd5fae-1e12-419c-a127-3366895815ee","Type":"ContainerDied","Data":"d9850ab9a8432dd94d929312929d1d027ecf04705f139b529bcebfd9b7afcffb"}
Nov 02 17:27:35.217931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.217869    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9850ab9a8432dd94d929312929d1d027ecf04705f139b529bcebfd9b7afcffb"
Nov 02 17:27:35.219847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.219826    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g" event={"ID":"944f2d06-866e-4a9b-82ee-eeac48f220b5","Type":"ContainerDied","Data":"5134e17d062610d884dbacfae1990c654fe1b0c207dc71554a1a6d3ee8b31bd5"}
Nov 02 17:27:35.219936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.219869    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65wc8g"
Nov 02 17:27:35.219936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.219870    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5134e17d062610d884dbacfae1990c654fe1b0c207dc71554a1a6d3ee8b31bd5"
Nov 02 17:27:35.352656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.352598    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:27:35.376426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.376402    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:27:35.503106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503027    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6\" (UniqueName: \"kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6\") pod \"44b9bfb7-ca6a-4350-a61f-970c21b84466\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") "
Nov 02 17:27:35.503106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503064    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util\") pod \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") "
Nov 02 17:27:35.503106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503098    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util\") pod \"44b9bfb7-ca6a-4350-a61f-970c21b84466\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") "
Nov 02 17:27:35.503385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503130    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb\" (UniqueName: \"kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb\") pod \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") "
Nov 02 17:27:35.503385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503159    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle\") pod \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\" (UID: \"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0\") "
Nov 02 17:27:35.503385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503197    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle\") pod \"44b9bfb7-ca6a-4350-a61f-970c21b84466\" (UID: \"44b9bfb7-ca6a-4350-a61f-970c21b84466\") "
Nov 02 17:27:35.503618 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503587    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle" pod "5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" (UID: "5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:35.504015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.503981    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle" pod "44b9bfb7-ca6a-4350-a61f-970c21b84466" (UID: "44b9bfb7-ca6a-4350-a61f-970c21b84466"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:35.505397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.505373    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb" pod "5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" (UID: "5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0"). InnerVolumeSpecName "kube-api-access-dnkrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:35.505488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.505417    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6" pod "44b9bfb7-ca6a-4350-a61f-970c21b84466" (UID: "44b9bfb7-ca6a-4350-a61f-970c21b84466"). InnerVolumeSpecName "kube-api-access-djkp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:35.509503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.509479    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util" pod "44b9bfb7-ca6a-4350-a61f-970c21b84466" (UID: "44b9bfb7-ca6a-4350-a61f-970c21b84466"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:35.511970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.511945    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util" pod "5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" (UID: "5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:35.604274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.604244    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:35.604274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.604271    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkrb\" (UniqueName: \"kubernetes.io/projected/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-kube-api-access-dnkrb\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:35.604441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.604282    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:35.604441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.604291    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b9bfb7-ca6a-4350-a61f-970c21b84466-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:35.604441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.604300    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkp6\" (UniqueName: \"kubernetes.io/projected/44b9bfb7-ca6a-4350-a61f-970c21b84466-kube-api-access-djkp6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:35.604441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:35.604308    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:36.226319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:36.226288    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n"
Nov 02 17:27:36.226319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:36.226301    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rvz4n" event={"ID":"5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0","Type":"ContainerDied","Data":"10463169a0b793c7a1d8f893e6a1582ba996b1c9278435ba0dbace2903a22e8a"}
Nov 02 17:27:36.226791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:36.226347    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10463169a0b793c7a1d8f893e6a1582ba996b1c9278435ba0dbace2903a22e8a"
Nov 02 17:27:36.228291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:36.228259    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv" event={"ID":"44b9bfb7-ca6a-4350-a61f-970c21b84466","Type":"ContainerDied","Data":"05f058f0e6bcd12e068079af171d0c95615f6409265e88909dabdfb2096e9a27"}
Nov 02 17:27:36.228403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:36.228301    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f058f0e6bcd12e068079af171d0c95615f6409265e88909dabdfb2096e9a27"
Nov 02 17:27:36.228403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:36.228306    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajx5nv"
Nov 02 17:27:38.677292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:38.677258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:38.878015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:38.877987    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:39.595881 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.595836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:39.755799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.755766    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"]
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756062    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756075    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756086    4476 cpu_manager.go:414] "containerMap: removing container" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756091    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756100    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756105    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756113    4476 cpu_manager.go:414] "containerMap: removing container" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756118    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756125    4476 cpu_manager.go:414] "containerMap: removing container" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756129    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756135    4476 cpu_manager.go:414] "containerMap: removing container" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="pull"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756139    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="pull"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756144    4476 cpu_manager.go:414] "containerMap: removing container" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756149    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756154    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="pull"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756158    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="pull"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756164    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756168    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756174    4476 cpu_manager.go:414] "containerMap: removing container" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756179    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="extract"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756186    4476 cpu_manager.go:414] "containerMap: removing container" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="util"
Nov 02 17:27:39.756183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756191    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756196    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756201    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756207    4476 cpu_manager.go:414] "containerMap: removing container" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756211    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756217    4476 cpu_manager.go:414] "containerMap: removing container" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756221    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756228    4476 cpu_manager.go:414] "containerMap: removing container" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756232    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756237    4476 cpu_manager.go:414] "containerMap: removing container" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756241    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756247    4476 cpu_manager.go:414] "containerMap: removing container" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756251    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756257    4476 cpu_manager.go:414] "containerMap: removing container" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756260    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756265    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756269    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756276    4476 cpu_manager.go:414] "containerMap: removing container" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756280    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756286    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756289    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756296    4476 cpu_manager.go:414] "containerMap: removing container" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756299    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756305    4476 cpu_manager.go:414] "containerMap: removing container" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756309    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756315    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756319    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756324    4476 cpu_manager.go:414] "containerMap: removing container" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756328    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756334    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756338    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756346    4476 cpu_manager.go:414] "containerMap: removing container" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756350    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756355    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756360    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756366    4476 cpu_manager.go:414] "containerMap: removing container" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756370    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756376    4476 cpu_manager.go:414] "containerMap: removing container" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756380    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756385    4476 cpu_manager.go:414] "containerMap: removing container" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756389    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756394    4476 cpu_manager.go:414] "containerMap: removing container" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756398    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756406    4476 cpu_manager.go:414] "containerMap: removing container" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756410    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756415    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756419    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756424    4476 cpu_manager.go:414] "containerMap: removing container" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756428    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="pull"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756436    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756440    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="util"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756480    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="815dbad9-2474-4494-bd57-bbf12fb45c59" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756487    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4652dd-95d0-4c4d-bf0d-e6eaecba08c0" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756493    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="147470ac-1dc2-4dfd-aa18-0afc2db22fa6" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756499    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fe04e4-b3d4-4e13-a470-8d6f7634fc58" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756505    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cde11c-19ba-444b-b8af-646a219b7fe0" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756510    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="83951b5e-cc98-4e6f-a219-1f5fa01b31dd" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756515    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee7fcc2-0d4d-466c-9d91-2d11144d2bb2" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756520    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa0dbcd-aacf-4b94-bd8d-6dbf07a47c73" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756526    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cd5fae-1e12-419c-a127-3366895815ee" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756530    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="944f2d06-866e-4a9b-82ee-eeac48f220b5" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756535    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c304518f-84c5-4ec3-ac17-f79233dd87d4" containerName="extract"
Nov 02 17:27:39.756814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.756540    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b9bfb7-ca6a-4350-a61f-970c21b84466" containerName="extract"
Nov 02 17:27:39.758910 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.758896    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:39.811424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.811392    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 17:27:39.811545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.811486    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 17:27:39.811545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.811509    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 17:27:39.811693 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.811681    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 17:27:39.861360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.861335    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-z8nm9\""
Nov 02 17:27:39.861360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.861354    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 17:27:39.875070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.875046    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"]
Nov 02 17:27:39.879886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.879862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:39.936807 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.936777    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:39.936932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.936827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5lf\" (UniqueName: \"kubernetes.io/projected/689fb8e4-505a-4a0c-9f68-1600086e4a11-kube-api-access-xs5lf\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:39.936932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.936912    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:39.937085 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.936948    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/689fb8e4-505a-4a0c-9f68-1600086e4a11-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:39.937085 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.936986    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/689fb8e4-505a-4a0c-9f68-1600086e4a11-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:39.937085 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:39.937043    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.037707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.037677    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.037873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.037720    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.037873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.037753    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5lf\" (UniqueName: \"kubernetes.io/projected/689fb8e4-505a-4a0c-9f68-1600086e4a11-kube-api-access-xs5lf\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.037873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.037790    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.037873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.037825    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/689fb8e4-505a-4a0c-9f68-1600086e4a11-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.038056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.037880    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/689fb8e4-505a-4a0c-9f68-1600086e4a11-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.038251 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.038226    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/689fb8e4-505a-4a0c-9f68-1600086e4a11-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.040399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.040374    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/689fb8e4-505a-4a0c-9f68-1600086e4a11-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.040506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.040443    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.040557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.040500    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.040603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.040583    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/689fb8e4-505a-4a0c-9f68-1600086e4a11-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.097122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.097090    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5lf\" (UniqueName: \"kubernetes.io/projected/689fb8e4-505a-4a0c-9f68-1600086e4a11-kube-api-access-xs5lf\") pod \"opendatahub-operator-controller-manager-89b98757-wgq6c\" (UID: \"689fb8e4-505a-4a0c-9f68-1600086e4a11\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.194893 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.194807    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:40.371111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.371076    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:40.426519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.426491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:40.912082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:40.912051    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"]
Nov 02 17:27:40.915895 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:27:40.915871    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689fb8e4_505a_4a0c_9f68_1600086e4a11.slice/crio-e13b6486e5ffdf9bfa18f8596f1b1f85b2d2051205c03fe55deeca933d55698a WatchSource:0}: Error finding container e13b6486e5ffdf9bfa18f8596f1b1f85b2d2051205c03fe55deeca933d55698a: Status 404 returned error can't find the container with id e13b6486e5ffdf9bfa18f8596f1b1f85b2d2051205c03fe55deeca933d55698a
Nov 02 17:27:41.176852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:41.176755    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:41.250874 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:41.250841    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c" event={"ID":"689fb8e4-505a-4a0c-9f68-1600086e4a11","Type":"ContainerStarted","Data":"e13b6486e5ffdf9bfa18f8596f1b1f85b2d2051205c03fe55deeca933d55698a"}
Nov 02 17:27:41.393790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:41.393759    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:42.053459 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.053422    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"]
Nov 02 17:27:42.056131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.056103    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.110550 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.110512    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-c72pw\""
Nov 02 17:27:42.153697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.153399    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbw4x\" (UniqueName: \"kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.153697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.153456    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.153697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.153495    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.160232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.160203    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 17:27:42.160384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.160338    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 17:27:42.162695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.162665    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"]
Nov 02 17:27:42.254221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.254189    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw4x\" (UniqueName: \"kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.254388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.254225    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.254388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.254255    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.254584 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.254566    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.254642 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.254613    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.312724 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.312658    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw4x\" (UniqueName: \"kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.332287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.332254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:42.371432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.371401    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:42.545932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.545894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:42.756535 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.756366    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"]
Nov 02 17:27:42.759333 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:27:42.759305    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98e0489_33ef_4d41_9970_06614751e342.slice/crio-3511868d0528490a83c686bbc1ab9a5443d133cd5ad6dcaeb6ce08ef8b777b35 WatchSource:0}: Error finding container 3511868d0528490a83c686bbc1ab9a5443d133cd5ad6dcaeb6ce08ef8b777b35: Status 404 returned error can't find the container with id 3511868d0528490a83c686bbc1ab9a5443d133cd5ad6dcaeb6ce08ef8b777b35
Nov 02 17:27:42.954529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:42.954493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:43.151968 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:43.151928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:43.262827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:43.262699    4476 generic.go:356] "Generic (PLEG): container finished" podID="d98e0489-33ef-4d41-9970-06614751e342" containerID="040e74aeeff088a0f3a270c3d29528a7b9f70982281ff35eca060b52de66415e" exitCode=0
Nov 02 17:27:43.262827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:43.262740    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45" event={"ID":"d98e0489-33ef-4d41-9970-06614751e342","Type":"ContainerDied","Data":"040e74aeeff088a0f3a270c3d29528a7b9f70982281ff35eca060b52de66415e"}
Nov 02 17:27:43.262827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:43.262789    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45" event={"ID":"d98e0489-33ef-4d41-9970-06614751e342","Type":"ContainerStarted","Data":"3511868d0528490a83c686bbc1ab9a5443d133cd5ad6dcaeb6ce08ef8b777b35"}
Nov 02 17:27:44.270691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:44.270658    4476 generic.go:356] "Generic (PLEG): container finished" podID="d98e0489-33ef-4d41-9970-06614751e342" containerID="1d20d202630fbf41af076ba7396efa3521517cebe93adcf16df0a0d2f60330ba" exitCode=0
Nov 02 17:27:44.271169 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:44.270747    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45" event={"ID":"d98e0489-33ef-4d41-9970-06614751e342","Type":"ContainerDied","Data":"1d20d202630fbf41af076ba7396efa3521517cebe93adcf16df0a0d2f60330ba"}
Nov 02 17:27:44.783428 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:44.783397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:45.026464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:45.026427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:45.279433 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:45.279348    4476 generic.go:356] "Generic (PLEG): container finished" podID="689fb8e4-505a-4a0c-9f68-1600086e4a11" containerID="109df84c3dbef31ebb83cf4e1025d328aa60478e2006fb9c6c4e1ddf5376cded" exitCode=0
Nov 02 17:27:45.279930 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:45.279423    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c" event={"ID":"689fb8e4-505a-4a0c-9f68-1600086e4a11","Type":"ContainerDied","Data":"109df84c3dbef31ebb83cf4e1025d328aa60478e2006fb9c6c4e1ddf5376cded"}
Nov 02 17:27:45.281783 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:45.281762    4476 generic.go:356] "Generic (PLEG): container finished" podID="d98e0489-33ef-4d41-9970-06614751e342" containerID="1937bf7debfd75c099fb91221ae81ce99eccbcd10f6fc4dc9c0c5b167b40c4ba" exitCode=0
Nov 02 17:27:45.281869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:45.281805    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45" event={"ID":"d98e0489-33ef-4d41-9970-06614751e342","Type":"ContainerDied","Data":"1937bf7debfd75c099fb91221ae81ce99eccbcd10f6fc4dc9c0c5b167b40c4ba"}
Nov 02 17:27:46.288872 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.288837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c" event={"ID":"689fb8e4-505a-4a0c-9f68-1600086e4a11","Type":"ContainerStarted","Data":"ed2a9d380b7a519cc3fe7814731ad237fa8a8732b18865c9eae2f78fb68df42a"}
Nov 02 17:27:46.289334 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.289026    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:46.413346 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.413321    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:46.431959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.431921    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c" podStartSLOduration=2.5617705170000002 podStartE2EDuration="7.431909103s" totalImagesPullingTime="4.441264756s" totalInitContainerRuntime="428.87383ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:46.429955007 +0000 UTC m=+297.270560941" watchObservedRunningTime="2026-11-02 17:27:46.431909103 +0000 UTC m=+297.272515036"
Nov 02 17:27:46.488904 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.488880    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util\") pod \"d98e0489-33ef-4d41-9970-06614751e342\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") "
Nov 02 17:27:46.489003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.488964    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle\") pod \"d98e0489-33ef-4d41-9970-06614751e342\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") "
Nov 02 17:27:46.489003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.488993    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x\" (UniqueName: \"kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x\") pod \"d98e0489-33ef-4d41-9970-06614751e342\" (UID: \"d98e0489-33ef-4d41-9970-06614751e342\") "
Nov 02 17:27:46.489432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.489399    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle" pod "d98e0489-33ef-4d41-9970-06614751e342" (UID: "d98e0489-33ef-4d41-9970-06614751e342"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:46.491239 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.491219    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x" pod "d98e0489-33ef-4d41-9970-06614751e342" (UID: "d98e0489-33ef-4d41-9970-06614751e342"). InnerVolumeSpecName "kube-api-access-rbw4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:27:46.495961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.495941    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util" pod "d98e0489-33ef-4d41-9970-06614751e342" (UID: "d98e0489-33ef-4d41-9970-06614751e342"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:27:46.590187 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.590107    4476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-bundle\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:46.590187 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.590137    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbw4x\" (UniqueName: \"kubernetes.io/projected/d98e0489-33ef-4d41-9970-06614751e342-kube-api-access-rbw4x\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:46.590187 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:46.590149    4476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d98e0489-33ef-4d41-9970-06614751e342-util\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:27:47.295064 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:47.295036    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45"
Nov 02 17:27:47.295064 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:47.295047    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b8h45" event={"ID":"d98e0489-33ef-4d41-9970-06614751e342","Type":"ContainerDied","Data":"3511868d0528490a83c686bbc1ab9a5443d133cd5ad6dcaeb6ce08ef8b777b35"}
Nov 02 17:27:47.295515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:47.295112    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3511868d0528490a83c686bbc1ab9a5443d133cd5ad6dcaeb6ce08ef8b777b35"
Nov 02 17:27:47.316259 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:27:47.316235    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98e0489_33ef_4d41_9970_06614751e342.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 17:27:48.300810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:48.300771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:49.375288 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:49.375262    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:27:49.375887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:49.375871    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:27:51.982848 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.982805    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"]
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983200    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="extract"
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983216    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="extract"
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983237    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="util"
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983244    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="util"
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983274    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="pull"
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983281    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="pull"
Nov 02 17:27:51.983343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.983342    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98e0489-33ef-4d41-9970-06614751e342" containerName="extract"
Nov 02 17:27:51.987278 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:51.987256    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.033347 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.033316    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dv6b\" (UniqueName: \"kubernetes.io/projected/6cd6b4da-8afe-4091-9dff-005687a59790-kube-api-access-6dv6b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zn92p\" (UID: \"6cd6b4da-8afe-4091-9dff-005687a59790\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.033456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.033383    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd6b4da-8afe-4091-9dff-005687a59790-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zn92p\" (UID: \"6cd6b4da-8afe-4091-9dff-005687a59790\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.040791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.040775    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 17:27:52.040961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.040945    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:27:52.041050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.041035    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-94k8d\""
Nov 02 17:27:52.093151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.093126    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"]
Nov 02 17:27:52.134548 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.134520    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dv6b\" (UniqueName: \"kubernetes.io/projected/6cd6b4da-8afe-4091-9dff-005687a59790-kube-api-access-6dv6b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zn92p\" (UID: \"6cd6b4da-8afe-4091-9dff-005687a59790\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.134725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.134580    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd6b4da-8afe-4091-9dff-005687a59790-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zn92p\" (UID: \"6cd6b4da-8afe-4091-9dff-005687a59790\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.134922 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.134904    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd6b4da-8afe-4091-9dff-005687a59790-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zn92p\" (UID: \"6cd6b4da-8afe-4091-9dff-005687a59790\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.191759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.191724    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dv6b\" (UniqueName: \"kubernetes.io/projected/6cd6b4da-8afe-4091-9dff-005687a59790-kube-api-access-6dv6b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zn92p\" (UID: \"6cd6b4da-8afe-4091-9dff-005687a59790\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.297103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.297009    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"
Nov 02 17:27:52.587151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.587039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:52.722991 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.722947    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p"]
Nov 02 17:27:52.726113 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:27:52.726088    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd6b4da_8afe_4091_9dff_005687a59790.slice/crio-6bb2f704beb5115a7fd658f76d806b14dbd112a0d8678b0aa0cd02282fbd47aa WatchSource:0}: Error finding container 6bb2f704beb5115a7fd658f76d806b14dbd112a0d8678b0aa0cd02282fbd47aa: Status 404 returned error can't find the container with id 6bb2f704beb5115a7fd658f76d806b14dbd112a0d8678b0aa0cd02282fbd47aa
Nov 02 17:27:52.728795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:52.728776    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:27:53.322007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:53.321975    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p" event={"ID":"6cd6b4da-8afe-4091-9dff-005687a59790","Type":"ContainerStarted","Data":"6bb2f704beb5115a7fd658f76d806b14dbd112a0d8678b0aa0cd02282fbd47aa"}
Nov 02 17:27:54.271380 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:54.271345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:54.479387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:54.479344    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:56.339170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:56.339070    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p" event={"ID":"6cd6b4da-8afe-4091-9dff-005687a59790","Type":"ContainerStarted","Data":"64fe5f3f364df9a414c261ab640b9ef60a7e2a8f5a5ea528ea92d1b217beb381"}
Nov 02 17:27:56.457709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:56.457652    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zn92p" podStartSLOduration=2.271225703 podStartE2EDuration="5.457618405s" totalImagesPullingTime="3.186392702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:56.45568253 +0000 UTC m=+307.296288461" watchObservedRunningTime="2026-11-02 17:27:56.457618405 +0000 UTC m=+307.298224339"
Nov 02 17:27:57.298457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:57.298421    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-wgq6c"
Nov 02 17:27:57.667498 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:57.667466    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:57.933124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:57.933045    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:58.133364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.133325    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"]
Nov 02 17:27:58.135747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.135729    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.185830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.185758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/967c923b-9582-4fc2-a950-9a45713f49b6-kube-api-access-88dcl\") pod \"cert-manager-webhook-5dbbbf75cc-5lxs6\" (UID: \"967c923b-9582-4fc2-a950-9a45713f49b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.185830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.185829    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/967c923b-9582-4fc2-a950-9a45713f49b6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5lxs6\" (UID: \"967c923b-9582-4fc2-a950-9a45713f49b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.198116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.198087    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 17:27:58.198219 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.198135    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 17:27:58.199469 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.199452    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ffm5r\""
Nov 02 17:27:58.210312 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.210286    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"]
Nov 02 17:27:58.286507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.286474    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/967c923b-9582-4fc2-a950-9a45713f49b6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5lxs6\" (UID: \"967c923b-9582-4fc2-a950-9a45713f49b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.286681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.286520    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/967c923b-9582-4fc2-a950-9a45713f49b6-kube-api-access-88dcl\") pod \"cert-manager-webhook-5dbbbf75cc-5lxs6\" (UID: \"967c923b-9582-4fc2-a950-9a45713f49b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.344082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.344054    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dcl\" (UniqueName: \"kubernetes.io/projected/967c923b-9582-4fc2-a950-9a45713f49b6-kube-api-access-88dcl\") pod \"cert-manager-webhook-5dbbbf75cc-5lxs6\" (UID: \"967c923b-9582-4fc2-a950-9a45713f49b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.346074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.346047    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/967c923b-9582-4fc2-a950-9a45713f49b6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5lxs6\" (UID: \"967c923b-9582-4fc2-a950-9a45713f49b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.460016 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.459943    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:27:58.834672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.834577    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"]
Nov 02 17:27:58.837342 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:27:58.837300    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod967c923b_9582_4fc2_a950_9a45713f49b6.slice/crio-dd204af9ba8d7375ef440195c13db6679b1fa07c2f92ddd84e6195edef90110e WatchSource:0}: Error finding container dd204af9ba8d7375ef440195c13db6679b1fa07c2f92ddd84e6195edef90110e: Status 404 returned error can't find the container with id dd204af9ba8d7375ef440195c13db6679b1fa07c2f92ddd84e6195edef90110e
Nov 02 17:27:58.879608 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:58.879557    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:27:59.058259 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:27:59.058222    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:27:59.354834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:27:59.354802    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6" event={"ID":"967c923b-9582-4fc2-a950-9a45713f49b6","Type":"ContainerStarted","Data":"dd204af9ba8d7375ef440195c13db6679b1fa07c2f92ddd84e6195edef90110e"}
Nov 02 17:28:00.105482 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:00.105451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:00.325833 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:00.325803    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:00.616230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:00.616157    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:04.257404 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.257369    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:04.538070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.537981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:04.547904 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.547878    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"]
Nov 02 17:28:04.551301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.551285    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:04.606031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.606008    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8b8fm\""
Nov 02 17:28:04.640397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.640358    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lx6p\" (UniqueName: \"kubernetes.io/projected/4c792d4f-f412-4cc5-8e8f-07c797c31015-kube-api-access-2lx6p\") pod \"authorino-operator-7b7bb4bc77-rjmsd\" (UID: \"4c792d4f-f412-4cc5-8e8f-07c797c31015\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:04.655773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.655747    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 17:28:04.655773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.655754    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 17:28:04.657996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.657976    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"]
Nov 02 17:28:04.741424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.741396    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lx6p\" (UniqueName: \"kubernetes.io/projected/4c792d4f-f412-4cc5-8e8f-07c797c31015-kube-api-access-2lx6p\") pod \"authorino-operator-7b7bb4bc77-rjmsd\" (UID: \"4c792d4f-f412-4cc5-8e8f-07c797c31015\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:04.798662 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.798581    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lx6p\" (UniqueName: \"kubernetes.io/projected/4c792d4f-f412-4cc5-8e8f-07c797c31015-kube-api-access-2lx6p\") pod \"authorino-operator-7b7bb4bc77-rjmsd\" (UID: \"4c792d4f-f412-4cc5-8e8f-07c797c31015\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:04.864170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:04.864143    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:05.237835 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:05.237690    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"]
Nov 02 17:28:05.240816 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:05.240790    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c792d4f_f412_4cc5_8e8f_07c797c31015.slice/crio-a14c0f54e249d456005984a32afe927459e08121c9fd10be42205940abeb1d43 WatchSource:0}: Error finding container a14c0f54e249d456005984a32afe927459e08121c9fd10be42205940abeb1d43: Status 404 returned error can't find the container with id a14c0f54e249d456005984a32afe927459e08121c9fd10be42205940abeb1d43
Nov 02 17:28:05.382889 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:05.382858    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd" event={"ID":"4c792d4f-f412-4cc5-8e8f-07c797c31015","Type":"ContainerStarted","Data":"a14c0f54e249d456005984a32afe927459e08121c9fd10be42205940abeb1d43"}
Nov 02 17:28:05.573791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:05.573689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:05.806286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:05.806246    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:06.036677 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:06.036619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:07.911155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:07.911113    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"]
Nov 02 17:28:07.916616 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:07.916584    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:07.968412 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:07.968367    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cw8\" (UniqueName: \"kubernetes.io/projected/c0d81e3f-d92e-4a06-adab-a62d740747a0-kube-api-access-52cw8\") pod \"dns-operator-controller-manager-8557cbdb84-hw69v\" (UID: \"c0d81e3f-d92e-4a06-adab-a62d740747a0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:07.973670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:07.971605    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 17:28:07.973865 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:07.973846    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"]
Nov 02 17:28:08.020607 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.020574    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jccjf\""
Nov 02 17:28:08.069155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.069126    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cw8\" (UniqueName: \"kubernetes.io/projected/c0d81e3f-d92e-4a06-adab-a62d740747a0-kube-api-access-52cw8\") pod \"dns-operator-controller-manager-8557cbdb84-hw69v\" (UID: \"c0d81e3f-d92e-4a06-adab-a62d740747a0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:08.136936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.136906    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cw8\" (UniqueName: \"kubernetes.io/projected/c0d81e3f-d92e-4a06-adab-a62d740747a0-kube-api-access-52cw8\") pod \"dns-operator-controller-manager-8557cbdb84-hw69v\" (UID: \"c0d81e3f-d92e-4a06-adab-a62d740747a0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:08.232130 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.232051    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:08.475975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.475915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:08.620399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.620368    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"]
Nov 02 17:28:08.623584 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:08.623552    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d81e3f_d92e_4a06_adab_a62d740747a0.slice/crio-d7fd189e9e2789c23577677096ac7ffe9a427b168c2a59be7a1aae75474c7ab5 WatchSource:0}: Error finding container d7fd189e9e2789c23577677096ac7ffe9a427b168c2a59be7a1aae75474c7ab5: Status 404 returned error can't find the container with id d7fd189e9e2789c23577677096ac7ffe9a427b168c2a59be7a1aae75474c7ab5
Nov 02 17:28:08.718275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:08.718236    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:09.266804 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:09.266774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:09.406519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:09.406479    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v" event={"ID":"c0d81e3f-d92e-4a06-adab-a62d740747a0","Type":"ContainerStarted","Data":"d7fd189e9e2789c23577677096ac7ffe9a427b168c2a59be7a1aae75474c7ab5"}
Nov 02 17:28:09.408027 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:09.408005    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd" event={"ID":"4c792d4f-f412-4cc5-8e8f-07c797c31015","Type":"ContainerStarted","Data":"38285ff370109b58870a0395eca282c0c08e8dac6b5ad530f7d0ab7730305b71"}
Nov 02 17:28:09.408121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:09.408051    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:09.522697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:09.522649    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd" podStartSLOduration=1.6701099560000001 podStartE2EDuration="5.522614809s" totalImagesPullingTime="3.852504853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:09.521119341 +0000 UTC m=+320.361725276" watchObservedRunningTime="2026-11-02 17:28:09.522614809 +0000 UTC m=+320.363220743"
Nov 02 17:28:09.894418 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:09.894383    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:10.365111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:10.365054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:11.418562 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:11.418526    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v" event={"ID":"c0d81e3f-d92e-4a06-adab-a62d740747a0","Type":"ContainerStarted","Data":"cb3ae180ede8a93bd79a2e6a7e7bcb5ed952af447495736c934e74d6fea12ad9"}
Nov 02 17:28:11.418995 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:11.418604    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:11.574912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:11.574806    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v" podStartSLOduration=1.928493181 podStartE2EDuration="4.574791609s" totalImagesPullingTime="2.646298428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:11.572698187 +0000 UTC m=+322.413304118" watchObservedRunningTime="2026-11-02 17:28:11.574791609 +0000 UTC m=+322.415397542"
Nov 02 17:28:12.931351 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:12.931319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:13.118940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:13.118905    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:13.942967 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:13.942934    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"]
Nov 02 17:28:13.946330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:13.946309    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.005356 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.005320    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4rzcw\""
Nov 02 17:28:14.020690 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.020658    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtg8g\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.020942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.020913    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrpt\" (UniqueName: \"kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt\") pod \"kuadrant-operator-controller-manager-678d87766f-wtg8g\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.057005 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.056969    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"]
Nov 02 17:28:14.121617 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.121577    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtg8g\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.121802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.121655    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrpt\" (UniqueName: \"kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt\") pod \"kuadrant-operator-controller-manager-678d87766f-wtg8g\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.121998 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.121977    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtg8g\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.183461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.183430    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrpt\" (UniqueName: \"kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt\") pod \"kuadrant-operator-controller-manager-678d87766f-wtg8g\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.258466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.258391    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:14.468206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.468175    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:14.741172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:14.741074    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"]
Nov 02 17:28:14.743957 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:14.743929    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2275c4f_e7d1_4aa0_8612_1a7e3d2e7fb6.slice/crio-62db9ed842492fb3c0d5e356099564a63b584e210d50c7be3922145641a81c66 WatchSource:0}: Error finding container 62db9ed842492fb3c0d5e356099564a63b584e210d50c7be3922145641a81c66: Status 404 returned error can't find the container with id 62db9ed842492fb3c0d5e356099564a63b584e210d50c7be3922145641a81c66
Nov 02 17:28:15.439043 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.439002    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g" event={"ID":"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6","Type":"ContainerStarted","Data":"62db9ed842492fb3c0d5e356099564a63b584e210d50c7be3922145641a81c66"}
Nov 02 17:28:15.567785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.567750    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:15.843745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.843665    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"]
Nov 02 17:28:15.848289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.848264    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:15.938739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.938704    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fqbh\" (UniqueName: \"kubernetes.io/projected/93dd06a4-ba2c-474a-8a39-c3e591131472-kube-api-access-7fqbh\") pod \"limitador-operator-controller-manager-5778bb4449-wg9xn\" (UID: \"93dd06a4-ba2c-474a-8a39-c3e591131472\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:15.949973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.949941    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vggjz\""
Nov 02 17:28:15.951984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:15.951962    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"]
Nov 02 17:28:16.039902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:16.039865    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fqbh\" (UniqueName: \"kubernetes.io/projected/93dd06a4-ba2c-474a-8a39-c3e591131472-kube-api-access-7fqbh\") pod \"limitador-operator-controller-manager-5778bb4449-wg9xn\" (UID: \"93dd06a4-ba2c-474a-8a39-c3e591131472\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:16.102498 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:16.102421    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fqbh\" (UniqueName: \"kubernetes.io/projected/93dd06a4-ba2c-474a-8a39-c3e591131472-kube-api-access-7fqbh\") pod \"limitador-operator-controller-manager-5778bb4449-wg9xn\" (UID: \"93dd06a4-ba2c-474a-8a39-c3e591131472\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:16.161458 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:16.161420    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:16.537210 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:16.537186    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"]
Nov 02 17:28:16.539125 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:16.539096    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dd06a4_ba2c_474a_8a39_c3e591131472.slice/crio-9a1fa59d622494a71d58b02187cb9b82ace76d136635b0749c9717f73b9b6d2b WatchSource:0}: Error finding container 9a1fa59d622494a71d58b02187cb9b82ace76d136635b0749c9717f73b9b6d2b: Status 404 returned error can't find the container with id 9a1fa59d622494a71d58b02187cb9b82ace76d136635b0749c9717f73b9b6d2b
Nov 02 17:28:17.452884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:17.452803    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn" event={"ID":"93dd06a4-ba2c-474a-8a39-c3e591131472","Type":"ContainerStarted","Data":"9a1fa59d622494a71d58b02187cb9b82ace76d136635b0749c9717f73b9b6d2b"}
Nov 02 17:28:17.468452 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:17.467966    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:17.660254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:17.660217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:17.931168 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:17.931103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:18.129089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:18.128996    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:18.684443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:18.684401    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:18.882491 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:18.882429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:19.173834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:19.173533    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:19.466598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:19.466559    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6" event={"ID":"967c923b-9582-4fc2-a950-9a45713f49b6","Type":"ContainerStarted","Data":"62947aa30c52892754d1ad2e5113c308f0272e1c41dc609af094df20eed6c036"}
Nov 02 17:28:19.466826 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:19.466719    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:28:19.531411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:19.531375    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:19.581327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:19.581279    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6" podStartSLOduration=1.4727515709999999 podStartE2EDuration="21.581263106s" totalImagesPullingTime="20.108511535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:19.579762645 +0000 UTC m=+330.420368579" watchObservedRunningTime="2026-11-02 17:28:19.581263106 +0000 UTC m=+330.421869039"
Nov 02 17:28:19.759607 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:19.759465    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:20.415003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:20.414980    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rjmsd"
Nov 02 17:28:20.474714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:20.474678    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn" event={"ID":"93dd06a4-ba2c-474a-8a39-c3e591131472","Type":"ContainerStarted","Data":"50d649fb702d46683b6b4745ea21b297138864fef691cf6659ea13202b694c9d"}
Nov 02 17:28:20.693780 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:20.693727    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn" podStartSLOduration=1.905193003 podStartE2EDuration="5.693710803s" totalImagesPullingTime="3.7885178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:20.692908962 +0000 UTC m=+331.533514897" watchObservedRunningTime="2026-11-02 17:28:20.693710803 +0000 UTC m=+331.534316739"
Nov 02 17:28:20.897117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:20.897087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:21.138950 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.138921    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:21.350976 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.350946    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:21.404133 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.399255    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"]
Nov 02 17:28:21.406540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.406522    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.459789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.459754    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qnjxd\""
Nov 02 17:28:21.474579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.474536    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"]
Nov 02 17:28:21.483896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.483859    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g" event={"ID":"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6","Type":"ContainerStarted","Data":"67b53cd7e142a73b679e518ce5cb86364311ac6676953f893e829945b7b6be25"}
Nov 02 17:28:21.484469 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.484439    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:21.484578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.484476    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:21.505465 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505438    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505483    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505507    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505586    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505780 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505742    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b3b731-576e-426f-9f35-7ddf92d288ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505780 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505775    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505866 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505856    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.505910 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.505879    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmw2\" (UniqueName: \"kubernetes.io/projected/42b3b731-576e-426f-9f35-7ddf92d288ab-kube-api-access-njmw2\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.506383 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.506357    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.607709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.607677    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b3b731-576e-426f-9f35-7ddf92d288ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.607911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.607727    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.607911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.607873    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.607915    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmw2\" (UniqueName: \"kubernetes.io/projected/42b3b731-576e-426f-9f35-7ddf92d288ab-kube-api-access-njmw2\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.607977    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608007    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608048    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608094    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608138    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608272    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608306    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608363    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608462    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b3b731-576e-426f-9f35-7ddf92d288ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.608553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.608496    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.610211 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.610170    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.610700 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.610680    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.672015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.671933    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:21.675096 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.675062    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b3b731-576e-426f-9f35-7ddf92d288ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.675661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.675619    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmw2\" (UniqueName: \"kubernetes.io/projected/42b3b731-576e-426f-9f35-7ddf92d288ab-kube-api-access-njmw2\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f78nkx\" (UID: \"42b3b731-576e-426f-9f35-7ddf92d288ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.718965 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.718938    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:21.736431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.736375    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g" podStartSLOduration=2.803186482 podStartE2EDuration="8.736358016s" totalImagesPullingTime="5.933171534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:21.73442476 +0000 UTC m=+332.575030685" watchObservedRunningTime="2026-11-02 17:28:21.736358016 +0000 UTC m=+332.576963950"
Nov 02 17:28:21.858934 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:21.858902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:22.144851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.144824    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"]
Nov 02 17:28:22.147222 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:22.147193    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b3b731_576e_426f_9f35_7ddf92d288ab.slice/crio-8f19422694c0563290f2c61f53dcb247def4de960fc4249e2e31ad99a1773a13 WatchSource:0}: Error finding container 8f19422694c0563290f2c61f53dcb247def4de960fc4249e2e31ad99a1773a13: Status 404 returned error can't find the container with id 8f19422694c0563290f2c61f53dcb247def4de960fc4249e2e31ad99a1773a13
Nov 02 17:28:22.149534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.149498    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:28:22.149648 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.149587    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:28:22.149718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.149655    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:28:22.425814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.425742    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hw69v"
Nov 02 17:28:22.490324 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.490291    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx" event={"ID":"42b3b731-576e-426f-9f35-7ddf92d288ab","Type":"ContainerStarted","Data":"3fe383bdcb70449d0b9f4281b41019998dc9f7a732fc56b2e6018211f27e484d"}
Nov 02 17:28:22.490497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.490348    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx" event={"ID":"42b3b731-576e-426f-9f35-7ddf92d288ab","Type":"ContainerStarted","Data":"8f19422694c0563290f2c61f53dcb247def4de960fc4249e2e31ad99a1773a13"}
Nov 02 17:28:22.659547 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.659500    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx" podStartSLOduration=1.659487114 podStartE2EDuration="1.659487114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:22.657963869 +0000 UTC m=+333.498569802" watchObservedRunningTime="2026-11-02 17:28:22.659487114 +0000 UTC m=+333.500093047"
Nov 02 17:28:22.719296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:22.719221    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:23.444529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:23.444495    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:23.724538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:23.724472    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:23.821915 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:23.821880    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:24.016294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:24.016219    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:24.500411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:24.500384    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:24.501455 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:24.501438    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f78nkx"
Nov 02 17:28:25.148981 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.148947    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"]
Nov 02 17:28:25.151559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.151545    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.168360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.168338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:25.207770 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.207695    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 17:28:25.208295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.207882    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 17:28:25.208383 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.208328    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cvtld\""
Nov 02 17:28:25.211245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.211212    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"]
Nov 02 17:28:25.343052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.343023    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.343052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.343055    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.343255 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.343076    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-kube-api-access-s7b4c\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.374906 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.374876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:25.444585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.444496    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.444585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.444540    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.444585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.444569    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-kube-api-access-s7b4c\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.445223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.445203    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.447099 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.447081    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.477238 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.477206    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5lxs6"
Nov 02 17:28:25.501083 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.501045    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/f4f28bac-e0b7-41dc-b254-0a3bdd5f997c-kube-api-access-s7b4c\") pod \"kuadrant-console-plugin-66c4fb575d-wlcdx\" (UID: \"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:25.571861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.571812    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:25.762923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:25.762849    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"
Nov 02 17:28:26.131506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:26.131483    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx"]
Nov 02 17:28:26.133619 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:26.133593    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4f28bac_e0b7_41dc_b254_0a3bdd5f997c.slice/crio-6852757c8f9be7e3c21c2bd41d4f291d6c9c66e7261933cbb071b4c7f84b4c5e WatchSource:0}: Error finding container 6852757c8f9be7e3c21c2bd41d4f291d6c9c66e7261933cbb071b4c7f84b4c5e: Status 404 returned error can't find the container with id 6852757c8f9be7e3c21c2bd41d4f291d6c9c66e7261933cbb071b4c7f84b4c5e
Nov 02 17:28:26.511309 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:26.511231    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx" event={"ID":"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c","Type":"ContainerStarted","Data":"6852757c8f9be7e3c21c2bd41d4f291d6c9c66e7261933cbb071b4c7f84b4c5e"}
Nov 02 17:28:27.017170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:27.017131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:27.261159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:27.261131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:28.275688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.274269    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9gjw9"]
Nov 02 17:28:28.278127 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.278101    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:28:28.331063 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.331016    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jm2sm\""
Nov 02 17:28:28.332173 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.332142    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9gjw9"]
Nov 02 17:28:28.372003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.371912    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wvr\" (UniqueName: \"kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr\") pod \"authorino-56c6c9ddd5-9gjw9\" (UID: \"e40c509a-0b85-4f8c-96fb-40cc6a545f11\") " pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:28:28.472548 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.472504    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wvr\" (UniqueName: \"kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr\") pod \"authorino-56c6c9ddd5-9gjw9\" (UID: \"e40c509a-0b85-4f8c-96fb-40cc6a545f11\") " pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:28:28.533603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.533529    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wvr\" (UniqueName: \"kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr\") pod \"authorino-56c6c9ddd5-9gjw9\" (UID: \"e40c509a-0b85-4f8c-96fb-40cc6a545f11\") " pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:28:28.594665 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.594609    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:28:28.793119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.793036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:28.975108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:28.974911    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9gjw9"]
Nov 02 17:28:28.977421 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:28:28.977387    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode40c509a_0b85_4f8c_96fb_40cc6a545f11.slice/crio-06dff0c18647dedd0f98bbae600f15c60bf68914e153c86d599b1e13e6a5bff3 WatchSource:0}: Error finding container 06dff0c18647dedd0f98bbae600f15c60bf68914e153c86d599b1e13e6a5bff3: Status 404 returned error can't find the container with id 06dff0c18647dedd0f98bbae600f15c60bf68914e153c86d599b1e13e6a5bff3
Nov 02 17:28:29.065163 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:28:29.065038    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:28:29.471247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:29.471210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:29.529086 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:29.529039    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9" event={"ID":"e40c509a-0b85-4f8c-96fb-40cc6a545f11","Type":"ContainerStarted","Data":"06dff0c18647dedd0f98bbae600f15c60bf68914e153c86d599b1e13e6a5bff3"}
Nov 02 17:28:29.683364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:29.683258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:32.493006 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:32.492975    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 17:28:32.493467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:32.493054    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-wg9xn"
Nov 02 17:28:32.549113 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:32.549072    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx" event={"ID":"f4f28bac-e0b7-41dc-b254-0a3bdd5f997c","Type":"ContainerStarted","Data":"da1377a3f69176e2f6319d2e90febc81032a0d57310a4f89a190ad3e6ca6a401"}
Nov 02 17:28:32.550672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:32.550622    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9" event={"ID":"e40c509a-0b85-4f8c-96fb-40cc6a545f11","Type":"ContainerStarted","Data":"a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7"}
Nov 02 17:28:32.897563 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:32.897507    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wlcdx" podStartSLOduration=1.901824526 podStartE2EDuration="7.897486836s" totalImagesPullingTime="5.99566231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:32.896396998 +0000 UTC m=+343.737002929" watchObservedRunningTime="2026-11-02 17:28:32.897486836 +0000 UTC m=+343.738092772"
Nov 02 17:28:33.052986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:33.052934    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9" podStartSLOduration=1.634105032 podStartE2EDuration="5.052920356s" totalImagesPullingTime="3.418815324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:33.052304887 +0000 UTC m=+343.892910818" watchObservedRunningTime="2026-11-02 17:28:33.052920356 +0000 UTC m=+343.893526290"
Nov 02 17:28:34.177420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:34.177391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:34.389199 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:34.389169    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:38.021644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:38.021605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:38.211640 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:38.211603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:38.522644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:38.522589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:38.726329 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:38.726291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:40.912863 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:40.912821    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:41.098236 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:41.098205    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:41.476839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:41.476800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:46.101570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:46.101536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:46.411110 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:46.411082    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:46.874521 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:46.874491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:47.080756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:47.080728    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:54.027396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:54.027354    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:54.262583 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:28:54.262554    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:28:59.057984 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:28:59.057941    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:29:08.800197 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:08.800163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:09.021533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:09.021502    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:09.231200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:09.231170    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:14.799879 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:14.799841    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:15.147945 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:15.147915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:15.370572 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:15.370537    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:18.288550 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:18.288515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:18.514091 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:18.514061    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:18.785686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:18.785654    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:26.446288 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:26.446248    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:29.058297 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:29:29.058264    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:29:40.535327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:40.535239    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:40.777363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:40.777322    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:40.995481 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:40.995456    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:41.198621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.198591    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:41.370744 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.370714    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ws6kb"]
Nov 02 17:29:41.373399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.373384    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.420603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.420570    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:41.430946 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.430918    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ws6kb"]
Nov 02 17:29:41.477043 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.477009    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 17:29:41.477193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.477047    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 17:29:41.477193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.477061    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 17:29:41.477193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.477054    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ktvc5\""
Nov 02 17:29:41.483745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.483721    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"]
Nov 02 17:29:41.486107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.486087    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.513230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.513207    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfrrd\" (UniqueName: \"kubernetes.io/projected/13454cdd-8e2d-4c3c-92cf-0c6adc13355f-kube-api-access-tfrrd\") pod \"kserve-controller-manager-776cc66d47-ws6kb\" (UID: \"13454cdd-8e2d-4c3c-92cf-0c6adc13355f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.513355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.513278    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13454cdd-8e2d-4c3c-92cf-0c6adc13355f-cert\") pod \"kserve-controller-manager-776cc66d47-ws6kb\" (UID: \"13454cdd-8e2d-4c3c-92cf-0c6adc13355f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.538487 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.538465    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 17:29:41.538830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.538490    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vm292\""
Nov 02 17:29:41.545364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.545344    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"]
Nov 02 17:29:41.597541 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.597512    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7w47j"]
Nov 02 17:29:41.600264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.600246    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.613891 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.613871    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6sjz\" (UniqueName: \"kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz\") pod \"llmisvc-controller-manager-869d5dcc5d-j78tm\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.614006 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.613918    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13454cdd-8e2d-4c3c-92cf-0c6adc13355f-cert\") pod \"kserve-controller-manager-776cc66d47-ws6kb\" (UID: \"13454cdd-8e2d-4c3c-92cf-0c6adc13355f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.614006 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.613992    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-j78tm\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.614126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.614038    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfrrd\" (UniqueName: \"kubernetes.io/projected/13454cdd-8e2d-4c3c-92cf-0c6adc13355f-kube-api-access-tfrrd\") pod \"kserve-controller-manager-776cc66d47-ws6kb\" (UID: \"13454cdd-8e2d-4c3c-92cf-0c6adc13355f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.616369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.616348    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13454cdd-8e2d-4c3c-92cf-0c6adc13355f-cert\") pod \"kserve-controller-manager-776cc66d47-ws6kb\" (UID: \"13454cdd-8e2d-4c3c-92cf-0c6adc13355f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.628747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.628684    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:41.653283 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.653263    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pbfcv\""
Nov 02 17:29:41.653368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.653330    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 17:29:41.660488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.660466    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7w47j"]
Nov 02 17:29:41.671416 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.671395    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfrrd\" (UniqueName: \"kubernetes.io/projected/13454cdd-8e2d-4c3c-92cf-0c6adc13355f-kube-api-access-tfrrd\") pod \"kserve-controller-manager-776cc66d47-ws6kb\" (UID: \"13454cdd-8e2d-4c3c-92cf-0c6adc13355f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.685135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.685118    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:41.715214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.715179    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-j78tm\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.715377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.715221    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76341d23-9cf8-4a8b-becb-16a32e0c9775-tls-certs\") pod \"model-serving-api-75bd89fdfc-7w47j\" (UID: \"76341d23-9cf8-4a8b-becb-16a32e0c9775\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.715377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.715267    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgsv\" (UniqueName: \"kubernetes.io/projected/76341d23-9cf8-4a8b-becb-16a32e0c9775-kube-api-access-hrgsv\") pod \"model-serving-api-75bd89fdfc-7w47j\" (UID: \"76341d23-9cf8-4a8b-becb-16a32e0c9775\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.715377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.715301    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6sjz\" (UniqueName: \"kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz\") pod \"llmisvc-controller-manager-869d5dcc5d-j78tm\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.718259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.718227    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-j78tm\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.719923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.719899    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-f9fdr"]
Nov 02 17:29:41.725649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.725117    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.776923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.776896    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6sjz\" (UniqueName: \"kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz\") pod \"llmisvc-controller-manager-869d5dcc5d-j78tm\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.779761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.779704    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 17:29:41.779901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.779758    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-69jdm\""
Nov 02 17:29:41.779901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.779810    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 17:29:41.783740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.783716    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-f9fdr"]
Nov 02 17:29:41.797060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.797041    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:41.816342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.816317    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-metrics-certs\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.816445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.816361    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76341d23-9cf8-4a8b-becb-16a32e0c9775-tls-certs\") pod \"model-serving-api-75bd89fdfc-7w47j\" (UID: \"76341d23-9cf8-4a8b-becb-16a32e0c9775\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.816500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.816461    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgsv\" (UniqueName: \"kubernetes.io/projected/76341d23-9cf8-4a8b-becb-16a32e0c9775-kube-api-access-hrgsv\") pod \"model-serving-api-75bd89fdfc-7w47j\" (UID: \"76341d23-9cf8-4a8b-becb-16a32e0c9775\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.816603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.816579    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xqv\" (UniqueName: \"kubernetes.io/projected/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-kube-api-access-p8xqv\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.816689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.816672    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-cert\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.819284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.819260    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76341d23-9cf8-4a8b-becb-16a32e0c9775-tls-certs\") pod \"model-serving-api-75bd89fdfc-7w47j\" (UID: \"76341d23-9cf8-4a8b-becb-16a32e0c9775\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.835711 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.835678    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:41.874304 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.874275    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgsv\" (UniqueName: \"kubernetes.io/projected/76341d23-9cf8-4a8b-becb-16a32e0c9775-kube-api-access-hrgsv\") pod \"model-serving-api-75bd89fdfc-7w47j\" (UID: \"76341d23-9cf8-4a8b-becb-16a32e0c9775\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.912361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.912293    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:41.917834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.917815    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xqv\" (UniqueName: \"kubernetes.io/projected/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-kube-api-access-p8xqv\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.917914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.917866    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-cert\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.917914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.917889    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-metrics-certs\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.920291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.920269    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-cert\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.920561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.920542    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-metrics-certs\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:41.978444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:41.978414    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xqv\" (UniqueName: \"kubernetes.io/projected/56c7fe1e-8ea8-454c-9ce4-2585f83c60b6-kube-api-access-p8xqv\") pod \"odh-model-controller-769875db56-f9fdr\" (UID: \"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6\") " pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:42.044420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.044385    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:42.064896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.064869    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ws6kb"]
Nov 02 17:29:42.065710 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:29:42.065689    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13454cdd_8e2d_4c3c_92cf_0c6adc13355f.slice/crio-222b77be53f5fa16cfc448ea2809c175ecdfc8e181b475982a0211eae7cf08fc WatchSource:0}: Error finding container 222b77be53f5fa16cfc448ea2809c175ecdfc8e181b475982a0211eae7cf08fc: Status 404 returned error can't find the container with id 222b77be53f5fa16cfc448ea2809c175ecdfc8e181b475982a0211eae7cf08fc
Nov 02 17:29:42.173682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.173661    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"]
Nov 02 17:29:42.175535 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:29:42.175508    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod42616aae_c084_4561_959d_2e4b95092afb.slice/crio-217f7848e5ffcb77a2235242394f23bd0f878ed688f56ddb2b860de73544fa75 WatchSource:0}: Error finding container 217f7848e5ffcb77a2235242394f23bd0f878ed688f56ddb2b860de73544fa75: Status 404 returned error can't find the container with id 217f7848e5ffcb77a2235242394f23bd0f878ed688f56ddb2b860de73544fa75
Nov 02 17:29:42.288181 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.288151    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7w47j"]
Nov 02 17:29:42.291137 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:29:42.291111    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76341d23_9cf8_4a8b_becb_16a32e0c9775.slice/crio-a012b72cdeabe4890adc87f7f881f94e532b23bbd5d25e6ab5533b287f850573 WatchSource:0}: Error finding container a012b72cdeabe4890adc87f7f881f94e532b23bbd5d25e6ab5533b287f850573: Status 404 returned error can't find the container with id a012b72cdeabe4890adc87f7f881f94e532b23bbd5d25e6ab5533b287f850573
Nov 02 17:29:42.423750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.423700    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-f9fdr"]
Nov 02 17:29:42.426127 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:29:42.426097    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c7fe1e_8ea8_454c_9ce4_2585f83c60b6.slice/crio-d34a0fd54a2a0f8b19c4145be101e952154e405cbb0aeaa641e67d573000290b WatchSource:0}: Error finding container d34a0fd54a2a0f8b19c4145be101e952154e405cbb0aeaa641e67d573000290b: Status 404 returned error can't find the container with id d34a0fd54a2a0f8b19c4145be101e952154e405cbb0aeaa641e67d573000290b
Nov 02 17:29:42.898318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.898284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:42.910806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.910777    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb" event={"ID":"13454cdd-8e2d-4c3c-92cf-0c6adc13355f","Type":"ContainerStarted","Data":"222b77be53f5fa16cfc448ea2809c175ecdfc8e181b475982a0211eae7cf08fc"}
Nov 02 17:29:42.911993 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.911968    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-f9fdr" event={"ID":"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6","Type":"ContainerStarted","Data":"d34a0fd54a2a0f8b19c4145be101e952154e405cbb0aeaa641e67d573000290b"}
Nov 02 17:29:42.913151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.913129    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j" event={"ID":"76341d23-9cf8-4a8b-becb-16a32e0c9775","Type":"ContainerStarted","Data":"a012b72cdeabe4890adc87f7f881f94e532b23bbd5d25e6ab5533b287f850573"}
Nov 02 17:29:42.914316 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:42.914299    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm" event={"ID":"42616aae-c084-4561-959d-2e4b95092afb","Type":"ContainerStarted","Data":"217f7848e5ffcb77a2235242394f23bd0f878ed688f56ddb2b860de73544fa75"}
Nov 02 17:29:45.945303 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:45.945266    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j" event={"ID":"76341d23-9cf8-4a8b-becb-16a32e0c9775","Type":"ContainerStarted","Data":"aea753fc112108e6cc230b3ab8cf39e866d9159e5097c94b665d6f503ff258f0"}
Nov 02 17:29:45.946042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:45.946026    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:46.953237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:46.953201    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb" event={"ID":"13454cdd-8e2d-4c3c-92cf-0c6adc13355f","Type":"ContainerStarted","Data":"ff7e80197e22ecf6ef0e172668f8c6d29659f1486536e327f81216033eeba274"}
Nov 02 17:29:46.953699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:46.953489    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:29:47.068669 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:47.068607    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb" podStartSLOduration=1.276437904 podStartE2EDuration="6.068593781s" totalImagesPullingTime="4.792155877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:47.067332573 +0000 UTC m=+417.907938506" watchObservedRunningTime="2026-11-02 17:29:47.068593781 +0000 UTC m=+417.909199715"
Nov 02 17:29:47.069603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:47.069570    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j" podStartSLOduration=2.592004184 podStartE2EDuration="6.069557911s" totalImagesPullingTime="3.477553727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:46.068937646 +0000 UTC m=+416.909543581" watchObservedRunningTime="2026-11-02 17:29:47.069557911 +0000 UTC m=+417.910163844"
Nov 02 17:29:47.960878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:47.960846    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-f9fdr" event={"ID":"56c7fe1e-8ea8-454c-9ce4-2585f83c60b6","Type":"ContainerStarted","Data":"004a46fe4bf851fb8fcd42a6ca3fecf06452067f08f616f6b8657e9307e33b74"}
Nov 02 17:29:47.961357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:47.960908    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:47.962531 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:47.962503    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm" event={"ID":"42616aae-c084-4561-959d-2e4b95092afb","Type":"ContainerStarted","Data":"f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54"}
Nov 02 17:29:47.962743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:47.962554    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:29:48.084302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:48.084249    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-f9fdr" podStartSLOduration=2.516638554 podStartE2EDuration="7.084236191s" totalImagesPullingTime="4.567597637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:48.082093122 +0000 UTC m=+418.922699053" watchObservedRunningTime="2026-11-02 17:29:48.084236191 +0000 UTC m=+418.924842125"
Nov 02 17:29:48.197530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:48.197462    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm" podStartSLOduration=2.376427988 podStartE2EDuration="7.197448246s" totalImagesPullingTime="4.821020258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:48.195397592 +0000 UTC m=+419.036003549" watchObservedRunningTime="2026-11-02 17:29:48.197448246 +0000 UTC m=+419.038054180"
Nov 02 17:29:48.201573 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:48.201508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:48.551003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:48.550971    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:48.762716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:48.762686    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:48.963409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:48.963356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.174019 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:49.173935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.389553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:49.389524    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.590238 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:49.590151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:50.616797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:50.616763    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:50.819277 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:50.819212    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:51.024875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:51.024797    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:57.966537 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:57.966506    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7w47j"
Nov 02 17:29:58.970989 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:29:58.970953    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-f9fdr"
Nov 02 17:29:59.083463 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:29:59.083427    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:30:00.190725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.190697    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"]
Nov 02 17:30:00.194351 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.194334    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.258761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.257878    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 17:30:00.258761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.258591    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:30:00.258761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.258728    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:30:00.267694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.267664    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"]
Nov 02 17:30:00.286109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.286071    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894010-68zqh\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.286109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.286111    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mk26\" (UniqueName: \"kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26\") pod \"maas-api-key-cleanup-29894010-68zqh\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.387155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.387119    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894010-68zqh\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.387155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.387152    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mk26\" (UniqueName: \"kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26\") pod \"maas-api-key-cleanup-29894010-68zqh\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.387841 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.387822    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894010-68zqh\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.463379 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.463299    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mk26\" (UniqueName: \"kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26\") pod \"maas-api-key-cleanup-29894010-68zqh\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.506955 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.506920    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:00.900056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:00.900028    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"]
Nov 02 17:30:00.902442 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:30:00.902417    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ca3be5_d976_4b6a_92fc_b5007cf6c64f.slice/crio-15794f79cac224041fbd65995f3c56e5e3bf9d761521bf7ffddfad0ff060079d WatchSource:0}: Error finding container 15794f79cac224041fbd65995f3c56e5e3bf9d761521bf7ffddfad0ff060079d: Status 404 returned error can't find the container with id 15794f79cac224041fbd65995f3c56e5e3bf9d761521bf7ffddfad0ff060079d
Nov 02 17:30:01.029277 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:01.029241    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" event={"ID":"03ca3be5-d976-4b6a-92fc-b5007cf6c64f","Type":"ContainerStarted","Data":"15794f79cac224041fbd65995f3c56e5e3bf9d761521bf7ffddfad0ff060079d"}
Nov 02 17:30:03.252257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.252218    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:03.439125 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.439085    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fn9pw"]
Nov 02 17:30:03.442968 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.442944    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:03.455776 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.455743    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:03.499409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.499372    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fn9pw"]
Nov 02 17:30:03.615884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.615857    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wxx\" (UniqueName: \"kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx\") pod \"authorino-6f6944bcdd-fn9pw\" (UID: \"509dddda-6884-44ed-b205-6ca075e6ac5a\") " pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:03.716673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.716607    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wxx\" (UniqueName: \"kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx\") pod \"authorino-6f6944bcdd-fn9pw\" (UID: \"509dddda-6884-44ed-b205-6ca075e6ac5a\") " pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:03.773238 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.773196    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fn9pw"]
Nov 02 17:30:03.773519 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:03.773493    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t8wxx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-fn9pw" podUID="509dddda-6884-44ed-b205-6ca075e6ac5a"
Nov 02 17:30:03.779543 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.779510    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wxx\" (UniqueName: \"kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx\") pod \"authorino-6f6944bcdd-fn9pw\" (UID: \"509dddda-6884-44ed-b205-6ca075e6ac5a\") " pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:03.803872 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.803834    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bdd94c7f5-cgvg8"]
Nov 02 17:30:03.807851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.807776    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:03.865747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.865713    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bdd94c7f5-cgvg8"]
Nov 02 17:30:03.918434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:03.918340    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsw4t\" (UniqueName: \"kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t\") pod \"authorino-7bdd94c7f5-cgvg8\" (UID: \"5262f864-0e0c-455c-aff2-4385b3361111\") " pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:04.020000 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.019852    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsw4t\" (UniqueName: \"kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t\") pod \"authorino-7bdd94c7f5-cgvg8\" (UID: \"5262f864-0e0c-455c-aff2-4385b3361111\") " pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:04.063965 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.063926    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:04.072326 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.072301    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:04.077815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.077786    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsw4t\" (UniqueName: \"kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t\") pod \"authorino-7bdd94c7f5-cgvg8\" (UID: \"5262f864-0e0c-455c-aff2-4385b3361111\") " pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:04.122728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.122691    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:04.221918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.221878    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx\" (UniqueName: \"kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx\") pod \"509dddda-6884-44ed-b205-6ca075e6ac5a\" (UID: \"509dddda-6884-44ed-b205-6ca075e6ac5a\") "
Nov 02 17:30:04.224651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.224590    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx" pod "509dddda-6884-44ed-b205-6ca075e6ac5a" (UID: "509dddda-6884-44ed-b205-6ca075e6ac5a"). InnerVolumeSpecName "kube-api-access-t8wxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:30:04.303261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.303229    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bdd94c7f5-cgvg8"]
Nov 02 17:30:04.323437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.323399    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wxx\" (UniqueName: \"kubernetes.io/projected/509dddda-6884-44ed-b205-6ca075e6ac5a-kube-api-access-t8wxx\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:04.332426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.332395    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b7d485bd5-vptcz"]
Nov 02 17:30:04.347614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.347578    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.393709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.393673    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b7d485bd5-vptcz"]
Nov 02 17:30:04.400534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.400505    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 17:30:04.509937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.509914    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bdd94c7f5-cgvg8"]
Nov 02 17:30:04.512322 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:30:04.512293    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5262f864_0e0c_455c_aff2_4385b3361111.slice/crio-c9919050cea4eb28e446658ad1ea0950b7e908e2f53171ccc8ea6bf2e8a4091e WatchSource:0}: Error finding container c9919050cea4eb28e446658ad1ea0950b7e908e2f53171ccc8ea6bf2e8a4091e: Status 404 returned error can't find the container with id c9919050cea4eb28e446658ad1ea0950b7e908e2f53171ccc8ea6bf2e8a4091e
Nov 02 17:30:04.525241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.525205    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert\") pod \"authorino-7b7d485bd5-vptcz\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") " pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.525360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.525255    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6llgv\" (UniqueName: \"kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv\") pod \"authorino-7b7d485bd5-vptcz\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") " pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.626068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.626015    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert\") pod \"authorino-7b7d485bd5-vptcz\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") " pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.626068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.626061    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6llgv\" (UniqueName: \"kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv\") pod \"authorino-7b7d485bd5-vptcz\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") " pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.629384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.629349    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert\") pod \"authorino-7b7d485bd5-vptcz\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") " pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.683450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.683416    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6llgv\" (UniqueName: \"kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv\") pod \"authorino-7b7d485bd5-vptcz\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") " pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:04.961595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:04.961555    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:30:05.073400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:05.073351    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8" event={"ID":"5262f864-0e0c-455c-aff2-4385b3361111","Type":"ContainerStarted","Data":"c9919050cea4eb28e446658ad1ea0950b7e908e2f53171ccc8ea6bf2e8a4091e"}
Nov 02 17:30:05.073588 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:05.073427    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fn9pw"
Nov 02 17:30:05.298765 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:05.298729    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fn9pw"]
Nov 02 17:30:05.304795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:05.304743    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fn9pw"]
Nov 02 17:30:05.352390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:05.352363    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b7d485bd5-vptcz"]
Nov 02 17:30:05.355189 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:30:05.355155    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4022f7b_bc32_4eae_bb79_ca6fdbd4a449.slice/crio-91e575e6d4f6fca9b85648279a35a643714a9550b4b945c635985e975272b9fa WatchSource:0}: Error finding container 91e575e6d4f6fca9b85648279a35a643714a9550b4b945c635985e975272b9fa: Status 404 returned error can't find the container with id 91e575e6d4f6fca9b85648279a35a643714a9550b4b945c635985e975272b9fa
Nov 02 17:30:05.384003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:05.383963    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509dddda-6884-44ed-b205-6ca075e6ac5a" path="/var/lib/kubelet/pods/509dddda-6884-44ed-b205-6ca075e6ac5a/volumes"
Nov 02 17:30:06.080972 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.080937    4476 generic.go:356] "Generic (PLEG): container finished" podID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerID="b4bbe5693e7fe8654663c739411f9b2bd386f9582da544ed7beaef76c7bcf32d" exitCode=60
Nov 02 17:30:06.081163 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.081026    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" event={"ID":"03ca3be5-d976-4b6a-92fc-b5007cf6c64f","Type":"ContainerDied","Data":"b4bbe5693e7fe8654663c739411f9b2bd386f9582da544ed7beaef76c7bcf32d"}
Nov 02 17:30:06.081296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.081280    4476 scope.go:122] "RemoveContainer" containerID="b4bbe5693e7fe8654663c739411f9b2bd386f9582da544ed7beaef76c7bcf32d"
Nov 02 17:30:06.082869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.082829    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8" event={"ID":"5262f864-0e0c-455c-aff2-4385b3361111","Type":"ContainerStarted","Data":"1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2"}
Nov 02 17:30:06.082996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.082908    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8" podUID="5262f864-0e0c-455c-aff2-4385b3361111" containerName="authorino" containerID="cri-o://1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2" gracePeriod=30
Nov 02 17:30:06.084390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.084365    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b7d485bd5-vptcz" event={"ID":"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449","Type":"ContainerStarted","Data":"91e575e6d4f6fca9b85648279a35a643714a9550b4b945c635985e975272b9fa"}
Nov 02 17:30:06.349035 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.348986    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8" podStartSLOduration=1.89587485 podStartE2EDuration="3.348972778s" totalImagesPullingTime="1.453097928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:06.347696123 +0000 UTC m=+437.188302057" watchObservedRunningTime="2026-11-02 17:30:06.348972778 +0000 UTC m=+437.189578711"
Nov 02 17:30:06.434057 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.434035    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:06.544990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.544905    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t\" (UniqueName: \"kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t\") pod \"5262f864-0e0c-455c-aff2-4385b3361111\" (UID: \"5262f864-0e0c-455c-aff2-4385b3361111\") "
Nov 02 17:30:06.547211 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.547178    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t" pod "5262f864-0e0c-455c-aff2-4385b3361111" (UID: "5262f864-0e0c-455c-aff2-4385b3361111"). InnerVolumeSpecName "kube-api-access-jsw4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:30:06.646676 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:06.646647    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsw4t\" (UniqueName: \"kubernetes.io/projected/5262f864-0e0c-455c-aff2-4385b3361111-kube-api-access-jsw4t\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:07.091299 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.091209    4476 generic.go:356] "Generic (PLEG): container finished" podID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerID="a6356ef6f3aca5a5107f9dbe9dca934e44a46e0c4768a371c2864cb12fd7426e" exitCode=60
Nov 02 17:30:07.091299 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.091283    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" event={"ID":"03ca3be5-d976-4b6a-92fc-b5007cf6c64f","Type":"ContainerDied","Data":"a6356ef6f3aca5a5107f9dbe9dca934e44a46e0c4768a371c2864cb12fd7426e"}
Nov 02 17:30:07.091560 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.091325    4476 scope.go:122] "RemoveContainer" containerID="b4bbe5693e7fe8654663c739411f9b2bd386f9582da544ed7beaef76c7bcf32d"
Nov 02 17:30:07.091615 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.091561    4476 scope.go:122] "RemoveContainer" containerID="a6356ef6f3aca5a5107f9dbe9dca934e44a46e0c4768a371c2864cb12fd7426e"
Nov 02 17:30:07.091854 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:07.091831    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894010-68zqh_odh-ai-gateway-infra(03ca3be5-d976-4b6a-92fc-b5007cf6c64f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f"
Nov 02 17:30:07.092717 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.092695    4476 generic.go:356] "Generic (PLEG): container finished" podID="5262f864-0e0c-455c-aff2-4385b3361111" containerID="1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2" exitCode=0
Nov 02 17:30:07.092824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.092750    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8"
Nov 02 17:30:07.092875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.092750    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8" event={"ID":"5262f864-0e0c-455c-aff2-4385b3361111","Type":"ContainerDied","Data":"1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2"}
Nov 02 17:30:07.092922 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.092880    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bdd94c7f5-cgvg8" event={"ID":"5262f864-0e0c-455c-aff2-4385b3361111","Type":"ContainerDied","Data":"c9919050cea4eb28e446658ad1ea0950b7e908e2f53171ccc8ea6bf2e8a4091e"}
Nov 02 17:30:07.094490 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.094469    4476 generic.go:356] "Generic (PLEG): container finished" podID="96738341-dadc-4422-8c87-4e0a5c634789" containerID="bb0721961fc214e4255f946a100603ba98731c86073dc1be72269f296159a7e9" exitCode=0
Nov 02 17:30:07.094621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.094530    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" event={"ID":"96738341-dadc-4422-8c87-4e0a5c634789","Type":"ContainerDied","Data":"bb0721961fc214e4255f946a100603ba98731c86073dc1be72269f296159a7e9"}
Nov 02 17:30:07.094953 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.094936    4476 scope.go:122] "RemoveContainer" containerID="bb0721961fc214e4255f946a100603ba98731c86073dc1be72269f296159a7e9"
Nov 02 17:30:07.096095 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.096060    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:30:07.096188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.096151    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:30:07.096248 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.096197    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:30:07.096445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.096419    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b7d485bd5-vptcz" event={"ID":"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449","Type":"ContainerStarted","Data":"efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718"}
Nov 02 17:30:07.106012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.105996    4476 scope.go:122] "RemoveContainer" containerID="1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2"
Nov 02 17:30:07.114392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.114368    4476 scope.go:122] "RemoveContainer" containerID="1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2"
Nov 02 17:30:07.114766 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:07.114749    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2\": container with ID starting with 1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2 not found: ID does not exist" containerID="1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2"
Nov 02 17:30:07.114866 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.114775    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2"} err="failed to get container status \"1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2\": rpc error: code = NotFound desc = could not find container \"1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2\": container with ID starting with 1d8a18a9db59bd8260a790d78cdbf71df2d1a29ea4e43ff29febaa6ec9b29af2 not found: ID does not exist"
Nov 02 17:30:07.319579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.319541    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bdd94c7f5-cgvg8"]
Nov 02 17:30:07.327315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.327281    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bdd94c7f5-cgvg8"]
Nov 02 17:30:07.384651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.384095    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5262f864-0e0c-455c-aff2-4385b3361111" path="/var/lib/kubelet/pods/5262f864-0e0c-455c-aff2-4385b3361111/volumes"
Nov 02 17:30:07.598696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.598594    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b7d485bd5-vptcz" podStartSLOduration=2.300849461 podStartE2EDuration="3.598571288s" totalImagesPullingTime="1.297721827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:07.572878097 +0000 UTC m=+438.413484034" watchObservedRunningTime="2026-11-02 17:30:07.598571288 +0000 UTC m=+438.439177258"
Nov 02 17:30:07.603683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.603642    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9gjw9"]
Nov 02 17:30:07.604071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.604039    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9" podUID="e40c509a-0b85-4f8c-96fb-40cc6a545f11" containerName="authorino" containerID="cri-o://a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7" gracePeriod=30
Nov 02 17:30:07.976048 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:07.975754    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:30:08.105544 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.105455    4476 scope.go:122] "RemoveContainer" containerID="a6356ef6f3aca5a5107f9dbe9dca934e44a46e0c4768a371c2864cb12fd7426e"
Nov 02 17:30:08.105778 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:08.105748    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894010-68zqh_odh-ai-gateway-infra(03ca3be5-d976-4b6a-92fc-b5007cf6c64f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f"
Nov 02 17:30:08.108741 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.108600    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" event={"ID":"96738341-dadc-4422-8c87-4e0a5c634789","Type":"ContainerStarted","Data":"944a0e67ecbc5cf0ec80db75da41671ee03f462b818f0502ad5603170d299d13"}
Nov 02 17:30:08.116051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.116019    4476 generic.go:356] "Generic (PLEG): container finished" podID="e40c509a-0b85-4f8c-96fb-40cc6a545f11" containerID="a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7" exitCode=0
Nov 02 17:30:08.116730 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.116713    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9"
Nov 02 17:30:08.116830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.116812    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9" event={"ID":"e40c509a-0b85-4f8c-96fb-40cc6a545f11","Type":"ContainerDied","Data":"a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7"}
Nov 02 17:30:08.116896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.116846    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9gjw9" event={"ID":"e40c509a-0b85-4f8c-96fb-40cc6a545f11","Type":"ContainerDied","Data":"06dff0c18647dedd0f98bbae600f15c60bf68914e153c86d599b1e13e6a5bff3"}
Nov 02 17:30:08.116896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.116867    4476 scope.go:122] "RemoveContainer" containerID="a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7"
Nov 02 17:30:08.128986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.128963    4476 scope.go:122] "RemoveContainer" containerID="a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7"
Nov 02 17:30:08.129342 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:08.129325    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7\": container with ID starting with a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7 not found: ID does not exist" containerID="a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7"
Nov 02 17:30:08.129405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.129353    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7"} err="failed to get container status \"a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7\": rpc error: code = NotFound desc = could not find container \"a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7\": container with ID starting with a32c549a59906e3dec496cb474947530820f896037095de2c7f2bea4956482c7 not found: ID does not exist"
Nov 02 17:30:08.176496 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.176453    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr\" (UniqueName: \"kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr\") pod \"e40c509a-0b85-4f8c-96fb-40cc6a545f11\" (UID: \"e40c509a-0b85-4f8c-96fb-40cc6a545f11\") "
Nov 02 17:30:08.178954 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.178917    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr" pod "e40c509a-0b85-4f8c-96fb-40cc6a545f11" (UID: "e40c509a-0b85-4f8c-96fb-40cc6a545f11"). InnerVolumeSpecName "kube-api-access-67wvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:30:08.279022 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.278980    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wvr\" (UniqueName: \"kubernetes.io/projected/e40c509a-0b85-4f8c-96fb-40cc6a545f11-kube-api-access-67wvr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:08.294253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.294213    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:30:08.294413 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.294395    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 02 17:30:08.294463 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.294444    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 02 17:30:08.536504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.536471    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9gjw9"]
Nov 02 17:30:08.556655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:08.554457    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9gjw9"]
Nov 02 17:30:09.294852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:09.294814    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 02 17:30:09.295029 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:09.294864    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 02 17:30:09.381644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:09.381602    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40c509a-0b85-4f8c-96fb-40cc6a545f11" path="/var/lib/kubelet/pods/e40c509a-0b85-4f8c-96fb-40cc6a545f11/volumes"
Nov 02 17:30:10.295047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:10.295011    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 02 17:30:10.295426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:10.295063    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 02 17:30:11.294108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:11.294069    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 02 17:30:11.294328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:11.294160    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 02 17:30:12.294744 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:12.294710    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 02 17:30:12.295172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:12.294780    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 02 17:30:13.298789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:13.298752    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:30:13.299275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:13.299170    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:30:13.299805 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:13.299787    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:30:17.967532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:17.967503    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ws6kb"
Nov 02 17:30:18.970869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:18.970841    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:30:19.378827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:19.378800    4476 scope.go:122] "RemoveContainer" containerID="a6356ef6f3aca5a5107f9dbe9dca934e44a46e0c4768a371c2864cb12fd7426e"
Nov 02 17:30:20.183432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:20.183400    4476 generic.go:356] "Generic (PLEG): container finished" podID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerID="1e55af3f69ecc9ca635d0e0aeda99fb5a06a4c9f5e3003d4b619bc2d5d08eaa7" exitCode=60
Nov 02 17:30:20.183831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:20.183466    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" event={"ID":"03ca3be5-d976-4b6a-92fc-b5007cf6c64f","Type":"ContainerDied","Data":"1e55af3f69ecc9ca635d0e0aeda99fb5a06a4c9f5e3003d4b619bc2d5d08eaa7"}
Nov 02 17:30:20.183831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:20.183520    4476 scope.go:122] "RemoveContainer" containerID="a6356ef6f3aca5a5107f9dbe9dca934e44a46e0c4768a371c2864cb12fd7426e"
Nov 02 17:30:20.183923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:20.183890    4476 scope.go:122] "RemoveContainer" containerID="1e55af3f69ecc9ca635d0e0aeda99fb5a06a4c9f5e3003d4b619bc2d5d08eaa7"
Nov 02 17:30:20.184125 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:20.184107    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894010-68zqh_odh-ai-gateway-infra(03ca3be5-d976-4b6a-92fc-b5007cf6c64f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f"
Nov 02 17:30:21.323911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.323881    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"]
Nov 02 17:30:21.467425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.467399    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:21.493593 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.493566    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26\" (UniqueName: \"kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26\") pod \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") "
Nov 02 17:30:21.493734 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.493669    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca\") pod \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\" (UID: \"03ca3be5-d976-4b6a-92fc-b5007cf6c64f\") "
Nov 02 17:30:21.494004 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.493961    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca" pod "03ca3be5-d976-4b6a-92fc-b5007cf6c64f" (UID: "03ca3be5-d976-4b6a-92fc-b5007cf6c64f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:30:21.494357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.494322    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:21.496067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.496046    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26" pod "03ca3be5-d976-4b6a-92fc-b5007cf6c64f" (UID: "03ca3be5-d976-4b6a-92fc-b5007cf6c64f"). InnerVolumeSpecName "kube-api-access-6mk26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:30:21.594868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:21.594791    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mk26\" (UniqueName: \"kubernetes.io/projected/03ca3be5-d976-4b6a-92fc-b5007cf6c64f-kube-api-access-6mk26\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:30:22.198011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:22.197983    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"
Nov 02 17:30:22.198190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:22.198018    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh" event={"ID":"03ca3be5-d976-4b6a-92fc-b5007cf6c64f","Type":"ContainerDied","Data":"15794f79cac224041fbd65995f3c56e5e3bf9d761521bf7ffddfad0ff060079d"}
Nov 02 17:30:22.198190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:22.198085    4476 scope.go:122] "RemoveContainer" containerID="1e55af3f69ecc9ca635d0e0aeda99fb5a06a4c9f5e3003d4b619bc2d5d08eaa7"
Nov 02 17:30:22.368120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:22.368085    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"]
Nov 02 17:30:22.417385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:22.417354    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894010-68zqh"]
Nov 02 17:30:23.381846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:23.381811    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" path="/var/lib/kubelet/pods/03ca3be5-d976-4b6a-92fc-b5007cf6c64f/volumes"
Nov 02 17:30:29.058390 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:29.058331    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:30:42.174581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:42.174546    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:42.422460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:42.422414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:48.839189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839154    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"]
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839504    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839515    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839524    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e40c509a-0b85-4f8c-96fb-40cc6a545f11" containerName="authorino"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839529    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40c509a-0b85-4f8c-96fb-40cc6a545f11" containerName="authorino"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839542    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5262f864-0e0c-455c-aff2-4385b3361111" containerName="authorino"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839547    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5262f864-0e0c-455c-aff2-4385b3361111" containerName="authorino"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839561    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839566    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839621    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5262f864-0e0c-455c-aff2-4385b3361111" containerName="authorino"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839645    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839653    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:30:48.839673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.839666    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40c509a-0b85-4f8c-96fb-40cc6a545f11" containerName="authorino"
Nov 02 17:30:48.841844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.841829    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:48.900145 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.900116    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"]
Nov 02 17:30:48.930375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.930338    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee2ee5b-9244-4eaf-8ed6-e72a261d01f7-cert\") pod \"llmisvc-controller-manager-6466c554d8-zt8p9\" (UID: \"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:48.930522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:48.930394    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jqg\" (UniqueName: \"kubernetes.io/projected/dee2ee5b-9244-4eaf-8ed6-e72a261d01f7-kube-api-access-m8jqg\") pod \"llmisvc-controller-manager-6466c554d8-zt8p9\" (UID: \"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:49.031701 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:49.031668    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee2ee5b-9244-4eaf-8ed6-e72a261d01f7-cert\") pod \"llmisvc-controller-manager-6466c554d8-zt8p9\" (UID: \"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:49.031701 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:49.031706    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jqg\" (UniqueName: \"kubernetes.io/projected/dee2ee5b-9244-4eaf-8ed6-e72a261d01f7-kube-api-access-m8jqg\") pod \"llmisvc-controller-manager-6466c554d8-zt8p9\" (UID: \"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:49.034269 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:49.034241    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee2ee5b-9244-4eaf-8ed6-e72a261d01f7-cert\") pod \"llmisvc-controller-manager-6466c554d8-zt8p9\" (UID: \"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:49.096204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:49.096128    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jqg\" (UniqueName: \"kubernetes.io/projected/dee2ee5b-9244-4eaf-8ed6-e72a261d01f7-kube-api-access-m8jqg\") pod \"llmisvc-controller-manager-6466c554d8-zt8p9\" (UID: \"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:49.152741 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:49.152707    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:49.526815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:49.526754    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"]
Nov 02 17:30:49.528994 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:30:49.528968    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee2ee5b_9244_4eaf_8ed6_e72a261d01f7.slice/crio-88fa2b7ee5f74b2ef0348b350ba94451ccc495d778232bdf02c662296e382192 WatchSource:0}: Error finding container 88fa2b7ee5f74b2ef0348b350ba94451ccc495d778232bdf02c662296e382192: Status 404 returned error can't find the container with id 88fa2b7ee5f74b2ef0348b350ba94451ccc495d778232bdf02c662296e382192
Nov 02 17:30:50.351854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:50.351815    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9" event={"ID":"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7","Type":"ContainerStarted","Data":"88fa2b7ee5f74b2ef0348b350ba94451ccc495d778232bdf02c662296e382192"}
Nov 02 17:30:51.358702 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:51.358663    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9" event={"ID":"dee2ee5b-9244-4eaf-8ed6-e72a261d01f7","Type":"ContainerStarted","Data":"b5a0cbcc812efbeabd615c9ff88ed1172ce612460fa516e9e6d079bfe054b102"}
Nov 02 17:30:51.359097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:51.358757    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:30:51.476483 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:51.476422    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9" podStartSLOduration=2.632816993 podStartE2EDuration="3.476403597s" totalImagesPullingTime="843.586604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:51.473655827 +0000 UTC m=+482.314261762" watchObservedRunningTime="2026-11-02 17:30:51.476403597 +0000 UTC m=+482.317009533"
Nov 02 17:30:53.397473 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:30:53.397437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:59.058408 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:30:59.058372    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:31:22.366931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.366847    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zt8p9"
Nov 02 17:31:22.535079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.535044    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"]
Nov 02 17:31:22.535344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.535288    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm" podUID="42616aae-c084-4561-959d-2e4b95092afb" containerName="manager" containerID="cri-o://f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54" gracePeriod=30
Nov 02 17:31:22.782747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.782725    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:31:22.923507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.923433    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert\" (UniqueName: \"kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert\") pod \"42616aae-c084-4561-959d-2e4b95092afb\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") "
Nov 02 17:31:22.923507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.923480    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz\" (UniqueName: \"kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz\") pod \"42616aae-c084-4561-959d-2e4b95092afb\" (UID: \"42616aae-c084-4561-959d-2e4b95092afb\") "
Nov 02 17:31:22.925925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.925898    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz" pod "42616aae-c084-4561-959d-2e4b95092afb" (UID: "42616aae-c084-4561-959d-2e4b95092afb"). InnerVolumeSpecName "kube-api-access-b6sjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:31:22.926030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:22.925960    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert" pod "42616aae-c084-4561-959d-2e4b95092afb" (UID: "42616aae-c084-4561-959d-2e4b95092afb"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:23.024230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.024196    4476 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42616aae-c084-4561-959d-2e4b95092afb-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:31:23.024230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.024227    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6sjz\" (UniqueName: \"kubernetes.io/projected/42616aae-c084-4561-959d-2e4b95092afb-kube-api-access-b6sjz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:31:23.522287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.522250    4476 generic.go:356] "Generic (PLEG): container finished" podID="42616aae-c084-4561-959d-2e4b95092afb" containerID="f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54" exitCode=0
Nov 02 17:31:23.522743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.522294    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"
Nov 02 17:31:23.522743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.522329    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm" event={"ID":"42616aae-c084-4561-959d-2e4b95092afb","Type":"ContainerDied","Data":"f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54"}
Nov 02 17:31:23.522743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.522379    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm" event={"ID":"42616aae-c084-4561-959d-2e4b95092afb","Type":"ContainerDied","Data":"217f7848e5ffcb77a2235242394f23bd0f878ed688f56ddb2b860de73544fa75"}
Nov 02 17:31:23.522743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.522400    4476 scope.go:122] "RemoveContainer" containerID="f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54"
Nov 02 17:31:23.533721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.533612    4476 scope.go:122] "RemoveContainer" containerID="f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54"
Nov 02 17:31:23.534023 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:31:23.534006    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54\": container with ID starting with f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54 not found: ID does not exist" containerID="f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54"
Nov 02 17:31:23.534079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.534032    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54"} err="failed to get container status \"f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54\": rpc error: code = NotFound desc = could not find container \"f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54\": container with ID starting with f57aad3f2ee20125a9c49e788085432a124a435396c4b78239bfbd72cabc2a54 not found: ID does not exist"
Nov 02 17:31:23.684832 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.684759    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"]
Nov 02 17:31:23.734499 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.734471    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-j78tm"]
Nov 02 17:31:23.929361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:23.929111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:24.121369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:24.121343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:25.381256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:25.381218    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42616aae-c084-4561-959d-2e4b95092afb" path="/var/lib/kubelet/pods/42616aae-c084-4561-959d-2e4b95092afb/volumes"
Nov 02 17:31:29.058073 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:31:29.058036    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:31:30.577216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:30.577183    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:30.761737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:30.761703    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:42.539588 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:42.539548    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:52.814612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:52.814558    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:53.351651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:53.351591    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:54.549692 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:54.549613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:54.741940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:54.741906    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:57.179381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:57.179340    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:57.361929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:31:57.361897    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:59.058777 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:31:59.058740    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:32:00.663371 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:00.663329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:07.877656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:07.877590    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:11.422365 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:11.422323    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:15.154694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:15.154646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:18.659274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:18.659234    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:22.968848 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:22.968804    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:27.911013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:27.910966    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:29.059240 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:32:29.059204    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:32:31.610789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:31.610705    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:42.870651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:42.870587    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:49.420818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:49.420777    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:32:49.423174 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:49.423151    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:32:56.624577 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:56.624543    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:58.944594 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.944559    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-685c598f54-4cmqn"]
Nov 02 17:32:58.944994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.944943    4476 cpu_manager.go:414] "containerMap: removing container" podUID="42616aae-c084-4561-959d-2e4b95092afb" containerName="manager"
Nov 02 17:32:58.944994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.944955    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42616aae-c084-4561-959d-2e4b95092afb" containerName="manager"
Nov 02 17:32:58.944994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.944967    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:32:58.944994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.944971    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:32:58.945119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.945052    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="42616aae-c084-4561-959d-2e4b95092afb" containerName="manager"
Nov 02 17:32:58.945119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.945060    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ca3be5-d976-4b6a-92fc-b5007cf6c64f" containerName="cleanup"
Nov 02 17:32:58.947083 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.947065    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:58.964904 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.964860    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67pn\" (UniqueName: \"kubernetes.io/projected/e027aa27-0b9b-4c21-8e23-54644e537f96-kube-api-access-r67pn\") pod \"authorino-685c598f54-4cmqn\" (UID: \"e027aa27-0b9b-4c21-8e23-54644e537f96\") " pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:58.965043 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:58.964981    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e027aa27-0b9b-4c21-8e23-54644e537f96-tls-cert\") pod \"authorino-685c598f54-4cmqn\" (UID: \"e027aa27-0b9b-4c21-8e23-54644e537f96\") " pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:59.003825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.003790    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-685c598f54-4cmqn"]
Nov 02 17:32:59.058331 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:32:59.058293    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:32:59.065806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.065778    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67pn\" (UniqueName: \"kubernetes.io/projected/e027aa27-0b9b-4c21-8e23-54644e537f96-kube-api-access-r67pn\") pod \"authorino-685c598f54-4cmqn\" (UID: \"e027aa27-0b9b-4c21-8e23-54644e537f96\") " pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:59.065973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.065872    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e027aa27-0b9b-4c21-8e23-54644e537f96-tls-cert\") pod \"authorino-685c598f54-4cmqn\" (UID: \"e027aa27-0b9b-4c21-8e23-54644e537f96\") " pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:59.068553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.068527    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e027aa27-0b9b-4c21-8e23-54644e537f96-tls-cert\") pod \"authorino-685c598f54-4cmqn\" (UID: \"e027aa27-0b9b-4c21-8e23-54644e537f96\") " pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:59.122522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.122488    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67pn\" (UniqueName: \"kubernetes.io/projected/e027aa27-0b9b-4c21-8e23-54644e537f96-kube-api-access-r67pn\") pod \"authorino-685c598f54-4cmqn\" (UID: \"e027aa27-0b9b-4c21-8e23-54644e537f96\") " pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:59.258294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.258208    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-685c598f54-4cmqn"
Nov 02 17:32:59.578945 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.578854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:59.635657 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.635568    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-685c598f54-4cmqn"]
Nov 02 17:32:59.638022 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:32:59.637983    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode027aa27_0b9b_4c21_8e23_54644e537f96.slice/crio-c1c4bd8bb1cac68275f5f6c14299e55717224a091b620ed00798d9e5897aa2ce WatchSource:0}: Error finding container c1c4bd8bb1cac68275f5f6c14299e55717224a091b620ed00798d9e5897aa2ce: Status 404 returned error can't find the container with id c1c4bd8bb1cac68275f5f6c14299e55717224a091b620ed00798d9e5897aa2ce
Nov 02 17:32:59.639350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.639325    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:32:59.784521 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:32:59.784466    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:00.074291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:00.074254    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-685c598f54-4cmqn" event={"ID":"e027aa27-0b9b-4c21-8e23-54644e537f96","Type":"ContainerStarted","Data":"c1c4bd8bb1cac68275f5f6c14299e55717224a091b620ed00798d9e5897aa2ce"}
Nov 02 17:33:01.085824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:01.085798    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:01.275491 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:01.275451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:02.090621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.090586    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-685c598f54-4cmqn" event={"ID":"e027aa27-0b9b-4c21-8e23-54644e537f96","Type":"ContainerStarted","Data":"7c1c9c247b64dd453b74b83edc2c05aafbb9ab59cbad08b337f5febd6769c2ea"}
Nov 02 17:33:02.244493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.244430    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-685c598f54-4cmqn" podStartSLOduration=2.8575403120000002 podStartE2EDuration="4.244411219s" totalImagesPullingTime="1.386870907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:02.243844049 +0000 UTC m=+613.084449985" watchObservedRunningTime="2026-11-02 17:33:02.244411219 +0000 UTC m=+613.085017530"
Nov 02 17:33:02.318580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.318545    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b7d485bd5-vptcz"]
Nov 02 17:33:02.318824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.318798    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7b7d485bd5-vptcz" podUID="d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" containerName="authorino" containerID="cri-o://efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718" gracePeriod=30
Nov 02 17:33:02.573499 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.573474    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:33:02.591785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.591755    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert\" (UniqueName: \"kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert\") pod \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") "
Nov 02 17:33:02.591970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.591806    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv\" (UniqueName: \"kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv\") pod \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\" (UID: \"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449\") "
Nov 02 17:33:02.594833 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.594755    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv" pod "d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" (UID: "d4022f7b-bc32-4eae-bb79-ca6fdbd4a449"). InnerVolumeSpecName "kube-api-access-6llgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:33:02.603698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.603671    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert" pod "d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" (UID: "d4022f7b-bc32-4eae-bb79-ca6fdbd4a449"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:33:02.702044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.701985    4476 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-tls-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:33:02.702044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:02.702017    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6llgv\" (UniqueName: \"kubernetes.io/projected/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449-kube-api-access-6llgv\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:33:03.098340 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.098300    4476 generic.go:356] "Generic (PLEG): container finished" podID="d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" containerID="efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718" exitCode=0
Nov 02 17:33:03.098840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.098345    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b7d485bd5-vptcz"
Nov 02 17:33:03.098840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.098390    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b7d485bd5-vptcz" event={"ID":"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449","Type":"ContainerDied","Data":"efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718"}
Nov 02 17:33:03.098840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.098439    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b7d485bd5-vptcz" event={"ID":"d4022f7b-bc32-4eae-bb79-ca6fdbd4a449","Type":"ContainerDied","Data":"91e575e6d4f6fca9b85648279a35a643714a9550b4b945c635985e975272b9fa"}
Nov 02 17:33:03.098840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.098458    4476 scope.go:122] "RemoveContainer" containerID="efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718"
Nov 02 17:33:03.110460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.110431    4476 scope.go:122] "RemoveContainer" containerID="efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718"
Nov 02 17:33:03.110776 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:33:03.110756    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718\": container with ID starting with efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718 not found: ID does not exist" containerID="efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718"
Nov 02 17:33:03.110860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.110784    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718"} err="failed to get container status \"efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718\": rpc error: code = NotFound desc = could not find container \"efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718\": container with ID starting with efbe096662a7cacddd458e817fd875eb8b99022bc53bda279009b2153f568718 not found: ID does not exist"
Nov 02 17:33:03.281653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.281238    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b7d485bd5-vptcz"]
Nov 02 17:33:03.285651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.285032    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b7d485bd5-vptcz"]
Nov 02 17:33:03.381160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:03.381117    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" path="/var/lib/kubelet/pods/d4022f7b-bc32-4eae-bb79-ca6fdbd4a449/volumes"
Nov 02 17:33:15.568793 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:15.568760    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:15.750302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:15.750269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:27.356084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:27.356053    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:29.058093 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:33:29.058053    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:33:31.341822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:31.341787    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:31.545035 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:31.544981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:31.795136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:31.795100    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:31.997436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:31.997388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:32.197440 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:32.197402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:32.413102 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:32.413070    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:32.625143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:32.625105    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:32.828491 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:32.828453    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:33.023873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:33.023794    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:33.206497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:33.206462    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:34.637256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:34.637226    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:34.821344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:34.821312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:36.140810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:36.140780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:36.343659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:36.343598    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:36.815104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:36.815073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:37.015891 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:37.015853    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:37.806481 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:37.806449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:38.012716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:38.012672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:38.393409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:38.393375    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:38.580359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:38.580323    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:38.769988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:38.769895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:39.351284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:39.351251    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:39.566995 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:39.566960    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:40.334899 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:40.334855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:40.517511 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:40.517468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:42.229175 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:42.229137    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:42.440177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:42.440145    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:42.642022 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:42.641990    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:42.851288 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:42.851250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:43.210087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:43.210050    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:43.406346 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:43.406312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:45.251391 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:45.251356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:45.433950 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:45.433918    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:46.386611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:46.386574    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:46.595519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:46.595485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:47.787728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:47.787690    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:47.985468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:47.985436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:48.147227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.147191    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"]
Nov 02 17:33:48.147561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.147549    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" containerName="authorino"
Nov 02 17:33:48.147606 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.147563    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" containerName="authorino"
Nov 02 17:33:48.147670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.147645    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4022f7b-bc32-4eae-bb79-ca6fdbd4a449" containerName="authorino"
Nov 02 17:33:48.150686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.150668    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.229824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.229784    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6td4\" (UniqueName: \"kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4\") pod \"payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.230033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.229953    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.254756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.252567    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aa5b913c-dockercfg-wdqf4\""
Nov 02 17:33:48.254756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.252678    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aa5b913c\""
Nov 02 17:33:48.256779 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.256753    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"]
Nov 02 17:33:48.331600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.331554    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6td4\" (UniqueName: \"kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4\") pod \"payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.331822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.331694    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.332414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.332389    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.388561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.388530    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6td4\" (UniqueName: \"kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4\") pod \"payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.463431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.463346    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:48.840734 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:48.840706    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"]
Nov 02 17:33:48.843416 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:33:48.843388    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcca463_3af6_40eb_a142_124fcb6dc4eb.slice/crio-c005bed347591ba87e47b54cdbc5c75dd854abcf557631a85e24abca9498101c WatchSource:0}: Error finding container c005bed347591ba87e47b54cdbc5c75dd854abcf557631a85e24abca9498101c: Status 404 returned error can't find the container with id c005bed347591ba87e47b54cdbc5c75dd854abcf557631a85e24abca9498101c
Nov 02 17:33:49.249338 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:49.249304    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:49.357809 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:49.357778    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s" event={"ID":"cbcca463-3af6-40eb-a142-124fcb6dc4eb","Type":"ContainerStarted","Data":"c005bed347591ba87e47b54cdbc5c75dd854abcf557631a85e24abca9498101c"}
Nov 02 17:33:50.340298 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.340217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:50.540071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.540040    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:50.698943 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.698906    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"]
Nov 02 17:33:50.702463 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.702440    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:50.755945 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.755887    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jqq\" (UniqueName: \"kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq\") pod \"payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:50.756140 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.756009    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:50.757152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.757123    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3eb29e6e\""
Nov 02 17:33:50.806421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.806386    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3eb29e6e-dockercfg-nkcf4\""
Nov 02 17:33:50.809126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.809083    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"]
Nov 02 17:33:50.857001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.856960    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:50.857178 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.857089    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqq\" (UniqueName: \"kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq\") pod \"payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:50.857845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.857822    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:50.914763 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:50.914732    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jqq\" (UniqueName: \"kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq\") pod \"payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:51.018104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:51.018014    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:51.408060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:51.408034    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"]
Nov 02 17:33:51.411190 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:33:51.411155    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbf950c_ab61_4323_b51f_4b470a1c96c2.slice/crio-48fc6140d8e471d65739632a132c57d815931c4063ebe466bdfcf416a6f71651 WatchSource:0}: Error finding container 48fc6140d8e471d65739632a132c57d815931c4063ebe466bdfcf416a6f71651: Status 404 returned error can't find the container with id 48fc6140d8e471d65739632a132c57d815931c4063ebe466bdfcf416a6f71651
Nov 02 17:33:52.377574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.377534    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr" event={"ID":"6dbf950c-ab61-4323-b51f-4b470a1c96c2","Type":"ContainerStarted","Data":"8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b"}
Nov 02 17:33:52.377775 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.377587    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr" event={"ID":"6dbf950c-ab61-4323-b51f-4b470a1c96c2","Type":"ContainerStarted","Data":"48fc6140d8e471d65739632a132c57d815931c4063ebe466bdfcf416a6f71651"}
Nov 02 17:33:52.377775 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.377644    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:33:52.379162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.379135    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s" event={"ID":"cbcca463-3af6-40eb-a142-124fcb6dc4eb","Type":"ContainerStarted","Data":"96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2"}
Nov 02 17:33:52.379294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.379194    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:33:52.492373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.492271    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr" podStartSLOduration=1.678843689 podStartE2EDuration="2.492256228s" totalImagesPullingTime="813.412539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:52.490441228 +0000 UTC m=+663.331047185" watchObservedRunningTime="2026-11-02 17:33:52.492256228 +0000 UTC m=+663.332862153"
Nov 02 17:33:52.607690 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:52.607577    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s" podStartSLOduration=1.8244079229999999 podStartE2EDuration="4.607548555s" totalImagesPullingTime="2.783140632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:52.603524289 +0000 UTC m=+663.444130222" watchObservedRunningTime="2026-11-02 17:33:52.607548555 +0000 UTC m=+663.448154537"
Nov 02 17:33:53.058015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:53.057979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:53.236864 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:53.236829    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:57.500615 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.500574    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:57.721177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.721137    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:57.866662 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.866614    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"]
Nov 02 17:33:57.870516 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.870492    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:57.923751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.923702    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-135f915e-dockercfg-5p8hj\""
Nov 02 17:33:57.931080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.931036    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88bjd\" (UniqueName: \"kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd\") pod \"payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:57.931262 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.931133    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:57.973220 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.973161    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-135f915e\""
Nov 02 17:33:57.975718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:57.975695    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"]
Nov 02 17:33:58.019243 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.019210    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"]
Nov 02 17:33:58.022743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.022719    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.032252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.031981    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:58.032252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.032164    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bjd\" (UniqueName: \"kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd\") pod \"payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:58.033320 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.033198    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:58.080685 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.080652    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"]
Nov 02 17:33:58.090208 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.090178    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88bjd\" (UniqueName: \"kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd\") pod \"payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:58.133399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.133315    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume\") pod \"payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") " pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.133399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.133382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfzs\" (UniqueName: \"kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs\") pod \"payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") " pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.184920 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.184881    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:58.234313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.234278    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume\") pod \"payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") " pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.234488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.234367    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfzs\" (UniqueName: \"kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs\") pod \"payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") " pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.235517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.235191    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume\") pod \"payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") " pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.292232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.292197    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfzs\" (UniqueName: \"kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs\") pod \"payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") " pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.317204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.317158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:58.341240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.341196    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:58.574765 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.574730    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"]
Nov 02 17:33:58.579679 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:33:58.579650    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cd9b55_5f53_460a_996d_c0f9200cae88.slice/crio-2fa447700eafd4266380b58e7175900c1f6e759899dec01c8b3a8125b0fbc42d WatchSource:0}: Error finding container 2fa447700eafd4266380b58e7175900c1f6e759899dec01c8b3a8125b0fbc42d: Status 404 returned error can't find the container with id 2fa447700eafd4266380b58e7175900c1f6e759899dec01c8b3a8125b0fbc42d
Nov 02 17:33:58.716484 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:58.716455    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"]
Nov 02 17:33:58.718511 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:33:58.718479    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981eff2e_eca9_4423_bb5e_0f43a3122ec6.slice/crio-a6dc134c35542d9d8de60f6a0313531bca7455cfc75cd27738ed6931fdc7b0b2 WatchSource:0}: Error finding container a6dc134c35542d9d8de60f6a0313531bca7455cfc75cd27738ed6931fdc7b0b2: Status 404 returned error can't find the container with id a6dc134c35542d9d8de60f6a0313531bca7455cfc75cd27738ed6931fdc7b0b2
Nov 02 17:33:59.060071 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:33:59.059982    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:33:59.421557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.421519    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv" event={"ID":"89cd9b55-5f53-460a-996d-c0f9200cae88","Type":"ContainerStarted","Data":"03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2"}
Nov 02 17:33:59.421815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.421575    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv" event={"ID":"89cd9b55-5f53-460a-996d-c0f9200cae88","Type":"ContainerStarted","Data":"2fa447700eafd4266380b58e7175900c1f6e759899dec01c8b3a8125b0fbc42d"}
Nov 02 17:33:59.421815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.421729    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:33:59.423246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.423221    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm" event={"ID":"981eff2e-eca9-4423-bb5e-0f43a3122ec6","Type":"ContainerStarted","Data":"e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268"}
Nov 02 17:33:59.423370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.423260    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm" event={"ID":"981eff2e-eca9-4423-bb5e-0f43a3122ec6","Type":"ContainerStarted","Data":"a6dc134c35542d9d8de60f6a0313531bca7455cfc75cd27738ed6931fdc7b0b2"}
Nov 02 17:33:59.423370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.423298    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:33:59.535652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.535570    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv" podStartSLOduration=2.535555774 podStartE2EDuration="2.535555774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:59.535069822 +0000 UTC m=+670.375675758" watchObservedRunningTime="2026-11-02 17:33:59.535555774 +0000 UTC m=+670.376161708"
Nov 02 17:33:59.649245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:33:59.649192    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm" podStartSLOduration=2.649177471 podStartE2EDuration="2.649177471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:59.648136406 +0000 UTC m=+670.488742339" watchObservedRunningTime="2026-11-02 17:33:59.649177471 +0000 UTC m=+670.489783405"
Nov 02 17:34:00.481358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:00.481274    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:00.655004 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:00.654974    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:03.278575 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:03.278542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:03.387960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:03.387927    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:34:03.388166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:03.388008    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:34:03.463811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:03.463775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:03.658091 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:03.658046    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:08.094455 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.094413    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"]
Nov 02 17:34:08.097087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.097062    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.150395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.150368    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-135f915e-serving-cert\""
Nov 02 17:34:08.199813 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.199772    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:34:08.199813 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.199779    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:34:08.200030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.200003    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 17:34:08.200030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.200009    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94n2w\""
Nov 02 17:34:08.202677 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.202658    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"]
Nov 02 17:34:08.224285 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.224255    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgvf\" (UniqueName: \"kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.224429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.224331    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.224469 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.224437    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.224503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.224470    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.325553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.325510    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.325748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.325561    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.325748 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:34:08.325674    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-serving-cert: secret "maas-api-e2e-derive-135f915e-serving-cert" not found
Nov 02 17:34:08.325748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.325703    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.325917 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:34:08.325768    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls podName:87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e nodeName:}" failed. No retries permitted until 2026-11-02 17:34:08.825747419 +0000 UTC m=+679.666353352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls") pod "maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" (UID: "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e") : secret "maas-api-e2e-derive-135f915e-serving-cert" not found
Nov 02 17:34:08.325917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.325805    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgvf\" (UniqueName: \"kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.326621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.326598    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.328289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.328269    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.341535 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.341506    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:08.382776 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.382748    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgvf\" (UniqueName: \"kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.529071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.529039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:08.739909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.739824    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:08.832226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.832181    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:08.834742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:08.834724    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\") pod \"maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:09.010754 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:09.010667    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:09.388191 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:09.388161    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"]
Nov 02 17:34:09.392664 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:34:09.392621    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87df5c6b_f6a0_42e6_a58b_f1ef1c053d7e.slice/crio-8070d20ee969c158a52e1fcda58710d19b8f767f22523178f4f1c61e524af36a WatchSource:0}: Error finding container 8070d20ee969c158a52e1fcda58710d19b8f767f22523178f4f1c61e524af36a: Status 404 returned error can't find the container with id 8070d20ee969c158a52e1fcda58710d19b8f767f22523178f4f1c61e524af36a
Nov 02 17:34:09.479533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:09.479498    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" event={"ID":"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e","Type":"ContainerStarted","Data":"8070d20ee969c158a52e1fcda58710d19b8f767f22523178f4f1c61e524af36a"}
Nov 02 17:34:10.431488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:10.431454    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:34:10.432003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:10.431889    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:34:12.506677 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:12.506613    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" event={"ID":"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e","Type":"ContainerStarted","Data":"56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2"}
Nov 02 17:34:12.507123 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:12.506697    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:12.631861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:12.631800    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" podStartSLOduration=1.679785842 podStartE2EDuration="4.63178105s" totalImagesPullingTime="2.951995208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:12.630399924 +0000 UTC m=+683.471005860" watchObservedRunningTime="2026-11-02 17:34:12.63178105 +0000 UTC m=+683.472386986"
Nov 02 17:34:13.501802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:13.501766    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:17.847596 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.847554    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"]
Nov 02 17:34:17.850778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.850756    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.905096 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.905060    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-97ba043b-openshift-default-dockercfg-c9rrg\""
Nov 02 17:34:17.922331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922297    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922338    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922354    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922374    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922492    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922552    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922664    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922889 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922745    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8s8\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.922889 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.922820    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:17.996180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:17.996138    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"]
Nov 02 17:34:18.024339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024309    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024348    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024370    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024399    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024425    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024449    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024493    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024545    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8s8\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024604    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.024929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.024791    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.025115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.025092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.025115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.025101    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.025246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.025208    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.025741 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.025705    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.027007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.026981    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.027552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.027530    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.100446 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.100373    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.101699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.101611    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8s8\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.167599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.167564    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:18.517846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.517819    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:34:18.533616 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.533561    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:18.600603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.600578    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"]
Nov 02 17:34:18.602281 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:34:18.602256    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29af9182_405e_455d_93dd_69bff5b0e13b.slice/crio-39d948bca2883fbdc9e822a8623c02ed78a84d49a732586a1ba523a16163ea98 WatchSource:0}: Error finding container 39d948bca2883fbdc9e822a8623c02ed78a84d49a732586a1ba523a16163ea98: Status 404 returned error can't find the container with id 39d948bca2883fbdc9e822a8623c02ed78a84d49a732586a1ba523a16163ea98
Nov 02 17:34:18.604581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.604552    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:34:18.604691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.604618    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:34:18.604691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:18.604682    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:34:19.568371 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:19.568332    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" event={"ID":"29af9182-405e-455d-93dd-69bff5b0e13b","Type":"ContainerStarted","Data":"3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133"}
Nov 02 17:34:19.568801 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:19.568398    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" event={"ID":"29af9182-405e-455d-93dd-69bff5b0e13b","Type":"ContainerStarted","Data":"39d948bca2883fbdc9e822a8623c02ed78a84d49a732586a1ba523a16163ea98"}
Nov 02 17:34:19.689440 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:19.689382    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podStartSLOduration=2.68936261 podStartE2EDuration="2.68936261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:19.686511726 +0000 UTC m=+690.527117687" watchObservedRunningTime="2026-11-02 17:34:19.68936261 +0000 UTC m=+690.529968545"
Nov 02 17:34:20.168777 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:20.168734    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:20.170372 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:20.170350    4476 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 17:34:20.170497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:20.170395    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 17:34:21.169021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:21.168981    4476 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 17:34:21.169428 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:21.169055    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 17:34:22.168237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:22.168201    4476 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 17:34:22.168442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:22.168291    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 17:34:23.168804 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:23.168764    4476 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 17:34:23.169223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:23.168838    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 17:34:23.762119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:23.762073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:24.169089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:24.169056    4476 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 17:34:24.169509 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:24.169124    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 17:34:25.172686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:25.172653    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:25.173112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:25.173077    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:25.173980 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:25.173957    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:34:29.059473 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:34:29.059436    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:34:29.221422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:29.221381    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:32.165846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:32.165815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:33.394053 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:33.394011    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"]
Nov 02 17:34:33.719054 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:33.718963    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:33.911193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:33.911158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:34.336068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:34.336031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:38.395742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.395681    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s" podUID="cbcca463-3af6-40eb-a142-124fcb6dc4eb" containerName="payload-pre-processing" containerID="cri-o://96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2" gracePeriod=25
Nov 02 17:34:38.658672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.658577    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:34:38.685917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.685879    4476 generic.go:356] "Generic (PLEG): container finished" podID="cbcca463-3af6-40eb-a142-124fcb6dc4eb" containerID="96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2" exitCode=0
Nov 02 17:34:38.686100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.685931    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"
Nov 02 17:34:38.686100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.685933    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s" event={"ID":"cbcca463-3af6-40eb-a142-124fcb6dc4eb","Type":"ContainerDied","Data":"96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2"}
Nov 02 17:34:38.686100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.685987    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s" event={"ID":"cbcca463-3af6-40eb-a142-124fcb6dc4eb","Type":"ContainerDied","Data":"c005bed347591ba87e47b54cdbc5c75dd854abcf557631a85e24abca9498101c"}
Nov 02 17:34:38.686100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.686019    4476 scope.go:122] "RemoveContainer" containerID="96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2"
Nov 02 17:34:38.695441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.695417    4476 scope.go:122] "RemoveContainer" containerID="96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2"
Nov 02 17:34:38.695837 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:34:38.695812    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2\": container with ID starting with 96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2 not found: ID does not exist" containerID="96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2"
Nov 02 17:34:38.695957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.695844    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2"} err="failed to get container status \"96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2\": rpc error: code = NotFound desc = could not find container \"96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2\": container with ID starting with 96650483458d2a5ab0b34e5712ad231d812a8397d24dcc5ebc37996f59ecd5d2 not found: ID does not exist"
Nov 02 17:34:38.728541 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.728497    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4\" (UniqueName: \"kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4\") pod \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") "
Nov 02 17:34:38.728773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.728580    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume\") pod \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\" (UID: \"cbcca463-3af6-40eb-a142-124fcb6dc4eb\") "
Nov 02 17:34:38.728972 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.728947    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume" pod "cbcca463-3af6-40eb-a142-124fcb6dc4eb" (UID: "cbcca463-3af6-40eb-a142-124fcb6dc4eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:38.731156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.731123    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4" pod "cbcca463-3af6-40eb-a142-124fcb6dc4eb" (UID: "cbcca463-3af6-40eb-a142-124fcb6dc4eb"). InnerVolumeSpecName "kube-api-access-b6td4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:38.829510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.829475    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6td4\" (UniqueName: \"kubernetes.io/projected/cbcca463-3af6-40eb-a142-124fcb6dc4eb-kube-api-access-b6td4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:38.829510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:38.829505    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcca463-3af6-40eb-a142-124fcb6dc4eb-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:39.105653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:39.105603    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"]
Nov 02 17:34:39.112231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:39.112199    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa5b913c-7f59995b6c-cjc4s"]
Nov 02 17:34:39.384595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:39.384565    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcca463-3af6-40eb-a142-124fcb6dc4eb" path="/var/lib/kubelet/pods/cbcca463-3af6-40eb-a142-124fcb6dc4eb/volumes"
Nov 02 17:34:39.410247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:39.410211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:41.801434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:41.801395    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:41.988779 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:41.988746    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:43.561691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:43.561653    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:43.883917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:43.883887    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:45.097091 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:45.097049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:47.529661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:47.529613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:47.852376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:47.852291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:48.049636 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:48.049604    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:48.314021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:48.313986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:48.795402 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:48.795363    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"]
Nov 02 17:34:49.138226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:49.138184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:49.337986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:49.337943    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:49.743836 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:49.743778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:49.992054 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:49.992015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:52.030797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:52.030736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:52.238370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:52.238336    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:52.427505 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:52.427464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:53.796146 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:53.796098    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr" podUID="6dbf950c-ab61-4323-b51f-4b470a1c96c2" containerName="payload-pre-processing" containerID="cri-o://8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b" gracePeriod=25
Nov 02 17:34:54.042456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.042432    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:34:54.180897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.180868    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume\") pod \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") "
Nov 02 17:34:54.181069 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.180904    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq\" (UniqueName: \"kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq\") pod \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\" (UID: \"6dbf950c-ab61-4323-b51f-4b470a1c96c2\") "
Nov 02 17:34:54.181243 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.181220    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume" pod "6dbf950c-ab61-4323-b51f-4b470a1c96c2" (UID: "6dbf950c-ab61-4323-b51f-4b470a1c96c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:54.183333 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.183308    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq" pod "6dbf950c-ab61-4323-b51f-4b470a1c96c2" (UID: "6dbf950c-ab61-4323-b51f-4b470a1c96c2"). InnerVolumeSpecName "kube-api-access-w2jqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:54.282400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.282360    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbf950c-ab61-4323-b51f-4b470a1c96c2-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:54.282400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.282392    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jqq\" (UniqueName: \"kubernetes.io/projected/6dbf950c-ab61-4323-b51f-4b470a1c96c2-kube-api-access-w2jqq\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:34:54.789954 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.789913    4476 generic.go:356] "Generic (PLEG): container finished" podID="6dbf950c-ab61-4323-b51f-4b470a1c96c2" containerID="8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b" exitCode=0
Nov 02 17:34:54.790162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.789958    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"
Nov 02 17:34:54.790162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.789998    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr" event={"ID":"6dbf950c-ab61-4323-b51f-4b470a1c96c2","Type":"ContainerDied","Data":"8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b"}
Nov 02 17:34:54.790162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.790052    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr" event={"ID":"6dbf950c-ab61-4323-b51f-4b470a1c96c2","Type":"ContainerDied","Data":"48fc6140d8e471d65739632a132c57d815931c4063ebe466bdfcf416a6f71651"}
Nov 02 17:34:54.790162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.790078    4476 scope.go:122] "RemoveContainer" containerID="8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b"
Nov 02 17:34:54.800152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.799939    4476 scope.go:122] "RemoveContainer" containerID="8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b"
Nov 02 17:34:54.800450 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:34:54.800235    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b\": container with ID starting with 8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b not found: ID does not exist" containerID="8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b"
Nov 02 17:34:54.800450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.800258    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b"} err="failed to get container status \"8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b\": rpc error: code = NotFound desc = could not find container \"8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b\": container with ID starting with 8dff11b937dfb3f8c64c173595231e21ebf7c0c15d3f4c87c11b22593bf4695b not found: ID does not exist"
Nov 02 17:34:54.956900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:54.956868    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"]
Nov 02 17:34:55.006323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.006287    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3eb29e6e-644f9bbbb7-z4njr"]
Nov 02 17:34:55.251962 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.251923    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:55.381837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.381803    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbf950c-ab61-4323-b51f-4b470a1c96c2" path="/var/lib/kubelet/pods/6dbf950c-ab61-4323-b51f-4b470a1c96c2/volumes"
Nov 02 17:34:55.703616 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.703571    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:55.786991 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.786960    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"]
Nov 02 17:34:55.787401 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.787388    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6dbf950c-ab61-4323-b51f-4b470a1c96c2" containerName="payload-pre-processing"
Nov 02 17:34:55.787444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.787405    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbf950c-ab61-4323-b51f-4b470a1c96c2" containerName="payload-pre-processing"
Nov 02 17:34:55.787444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.787431    4476 cpu_manager.go:414] "containerMap: removing container" podUID="cbcca463-3af6-40eb-a142-124fcb6dc4eb" containerName="payload-pre-processing"
Nov 02 17:34:55.787444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.787436    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcca463-3af6-40eb-a142-124fcb6dc4eb" containerName="payload-pre-processing"
Nov 02 17:34:55.787534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.787499    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcca463-3af6-40eb-a142-124fcb6dc4eb" containerName="payload-pre-processing"
Nov 02 17:34:55.787534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.787508    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbf950c-ab61-4323-b51f-4b470a1c96c2" containerName="payload-pre-processing"
Nov 02 17:34:55.790716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.790689    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:55.843033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.842997    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a099c3\""
Nov 02 17:34:55.843494 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.843439    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a099c3-dockercfg-gbk7h\""
Nov 02 17:34:55.895393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.895354    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"]
Nov 02 17:34:55.898166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.898142    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:55.898256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.898202    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjvs\" (UniqueName: \"kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs\") pod \"payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:55.912786 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.912749    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:55.998933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.998844    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjvs\" (UniqueName: \"kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs\") pod \"payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:55.998933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.998931    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:55.999550 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:55.999532    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:56.077998 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.077967    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjvs\" (UniqueName: \"kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs\") pod \"payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:56.104566 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.104526    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:56.189692 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.189654    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:56.489691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.489656    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"]
Nov 02 17:34:56.792444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.792344    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:56.811661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.809495    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" event={"ID":"844410f3-6b62-40c3-af5d-3dfcda2c7977","Type":"ContainerStarted","Data":"711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839"}
Nov 02 17:34:56.811661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.809566    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" event={"ID":"844410f3-6b62-40c3-af5d-3dfcda2c7977","Type":"ContainerStarted","Data":"26a399c86bef4463615f1974eac0eaf85c8178a46ee731b8256f95159e20d648"}
Nov 02 17:34:56.811661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.809797    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:34:56.923559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.923508    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" podStartSLOduration=1.923492755 podStartE2EDuration="1.923492755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:56.922521618 +0000 UTC m=+727.763127552" watchObservedRunningTime="2026-11-02 17:34:56.923492755 +0000 UTC m=+727.764098689"
Nov 02 17:34:56.985406 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:56.985373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:58.820387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:34:58.820348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:59.058673 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:34:59.058611    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:35:00.993177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:00.993142    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"]
Nov 02 17:35:01.103360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:01.103328    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"]
Nov 02 17:35:01.420672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:01.420603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:01.645242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:01.645202    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:03.214105 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.214071    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:03.216285 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.216261    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"]
Nov 02 17:35:03.219857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.219841    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.324729 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.324695    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"]
Nov 02 17:35:03.375582 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375540    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.375582 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375578    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.375870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375612    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.375870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375781    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.375870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375830    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw5c\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.376051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375962    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.376051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.375992    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.376051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.376016    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.376212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.376109    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477465    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477513    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477537    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477705    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477757    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477788    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477832    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477884    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.477925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477924    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw5c\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.478240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.477980    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.478240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.478000    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.478240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.478219    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.478562 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.478267    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.478562 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.478469    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.480085 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.480066    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.480426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.480405    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.534904 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.534875    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.535060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.535018    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw5c\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:03.832494 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:03.832405    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:04.264109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.264074    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"]
Nov 02 17:35:04.266370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:04.266338    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a57c34_e0f6_4ca4_877f_012a21f4535e.slice/crio-a27a5bc44d0d72869bbcbb19a456a79c0fe56d90283e1d5f70a3c86292506633 WatchSource:0}: Error finding container a27a5bc44d0d72869bbcbb19a456a79c0fe56d90283e1d5f70a3c86292506633: Status 404 returned error can't find the container with id a27a5bc44d0d72869bbcbb19a456a79c0fe56d90283e1d5f70a3c86292506633
Nov 02 17:35:04.269319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.269284    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:35:04.269411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.269359    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:35:04.269411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.269391    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:35:04.778675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.778615    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:04.860171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.860134    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6" event={"ID":"d6a57c34-e0f6-4ca4-877f-012a21f4535e","Type":"ContainerStarted","Data":"0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7"}
Nov 02 17:35:04.860394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.860195    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6" event={"ID":"d6a57c34-e0f6-4ca4-877f-012a21f4535e","Type":"ContainerStarted","Data":"a27a5bc44d0d72869bbcbb19a456a79c0fe56d90283e1d5f70a3c86292506633"}
Nov 02 17:35:04.974933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.974890    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:04.993838 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:04.993785    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6" podStartSLOduration=1.993769784 podStartE2EDuration="1.993769784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:04.991689456 +0000 UTC m=+735.832295390" watchObservedRunningTime="2026-11-02 17:35:04.993769784 +0000 UTC m=+735.834375718"
Nov 02 17:35:05.191050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.191017    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:05.409733 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.409694    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:05.833021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.832983    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:05.838228 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.838204    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:05.866167 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.866126    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:05.867228 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.867207    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:05.897931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.897895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:05.993994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:05.993946    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm" podUID="981eff2e-eca9-4423-bb5e-0f43a3122ec6" containerName="payload-processing" containerID="cri-o://e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268" gracePeriod=25
Nov 02 17:35:06.104075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.103985    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv" podUID="89cd9b55-5f53-460a-996d-c0f9200cae88" containerName="payload-pre-processing" containerID="cri-o://03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2" gracePeriod=25
Nov 02 17:35:06.259751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.259725    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:35:06.366215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.366192    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:35:06.408144 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.408108    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs\" (UniqueName: \"kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs\") pod \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") "
Nov 02 17:35:06.408330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.408198    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume\") pod \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\" (UID: \"981eff2e-eca9-4423-bb5e-0f43a3122ec6\") "
Nov 02 17:35:06.408912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.408880    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume" pod "981eff2e-eca9-4423-bb5e-0f43a3122ec6" (UID: "981eff2e-eca9-4423-bb5e-0f43a3122ec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:06.410726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.410704    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs" pod "981eff2e-eca9-4423-bb5e-0f43a3122ec6" (UID: "981eff2e-eca9-4423-bb5e-0f43a3122ec6"). InnerVolumeSpecName "kube-api-access-2qfzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:06.509342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.509314    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd\" (UniqueName: \"kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd\") pod \"89cd9b55-5f53-460a-996d-c0f9200cae88\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") "
Nov 02 17:35:06.509511 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.509355    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume\") pod \"89cd9b55-5f53-460a-996d-c0f9200cae88\" (UID: \"89cd9b55-5f53-460a-996d-c0f9200cae88\") "
Nov 02 17:35:06.509619 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.509600    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfzs\" (UniqueName: \"kubernetes.io/projected/981eff2e-eca9-4423-bb5e-0f43a3122ec6-kube-api-access-2qfzs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:06.509692 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.509642    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981eff2e-eca9-4423-bb5e-0f43a3122ec6-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:06.509794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.509776    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume" pod "89cd9b55-5f53-460a-996d-c0f9200cae88" (UID: "89cd9b55-5f53-460a-996d-c0f9200cae88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:06.511692 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.511670    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd" pod "89cd9b55-5f53-460a-996d-c0f9200cae88" (UID: "89cd9b55-5f53-460a-996d-c0f9200cae88"). InnerVolumeSpecName "kube-api-access-88bjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:06.610782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.610755    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88bjd\" (UniqueName: \"kubernetes.io/projected/89cd9b55-5f53-460a-996d-c0f9200cae88-kube-api-access-88bjd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:06.610782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.610784    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89cd9b55-5f53-460a-996d-c0f9200cae88-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:06.716198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.716078    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:06.874186 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.874097    4476 generic.go:356] "Generic (PLEG): container finished" podID="89cd9b55-5f53-460a-996d-c0f9200cae88" containerID="03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2" exitCode=0
Nov 02 17:35:06.874186 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.874155    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"
Nov 02 17:35:06.874429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.874190    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv" event={"ID":"89cd9b55-5f53-460a-996d-c0f9200cae88","Type":"ContainerDied","Data":"03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2"}
Nov 02 17:35:06.874429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.874255    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv" event={"ID":"89cd9b55-5f53-460a-996d-c0f9200cae88","Type":"ContainerDied","Data":"2fa447700eafd4266380b58e7175900c1f6e759899dec01c8b3a8125b0fbc42d"}
Nov 02 17:35:06.874429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.874284    4476 scope.go:122] "RemoveContainer" containerID="03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2"
Nov 02 17:35:06.875829 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.875812    4476 generic.go:356] "Generic (PLEG): container finished" podID="981eff2e-eca9-4423-bb5e-0f43a3122ec6" containerID="e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268" exitCode=0
Nov 02 17:35:06.875912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.875866    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"
Nov 02 17:35:06.876670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.876646    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm" event={"ID":"981eff2e-eca9-4423-bb5e-0f43a3122ec6","Type":"ContainerDied","Data":"e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268"}
Nov 02 17:35:06.876782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.876697    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm" event={"ID":"981eff2e-eca9-4423-bb5e-0f43a3122ec6","Type":"ContainerDied","Data":"a6dc134c35542d9d8de60f6a0313531bca7455cfc75cd27738ed6931fdc7b0b2"}
Nov 02 17:35:06.885515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.885493    4476 scope.go:122] "RemoveContainer" containerID="03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2"
Nov 02 17:35:06.885920 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:06.885900    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2\": container with ID starting with 03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2 not found: ID does not exist" containerID="03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2"
Nov 02 17:35:06.886015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.885934    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2"} err="failed to get container status \"03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2\": rpc error: code = NotFound desc = could not find container \"03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2\": container with ID starting with 03656f86b4a5755d51a0d7010f030dfaf068c495ef046a9e2409f4f181d90ab2 not found: ID does not exist"
Nov 02 17:35:06.886015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.885959    4476 scope.go:122] "RemoveContainer" containerID="e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268"
Nov 02 17:35:06.896953 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.896930    4476 scope.go:122] "RemoveContainer" containerID="e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268"
Nov 02 17:35:06.897348 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:06.897324    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268\": container with ID starting with e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268 not found: ID does not exist" containerID="e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268"
Nov 02 17:35:06.897467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:06.897359    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268"} err="failed to get container status \"e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268\": rpc error: code = NotFound desc = could not find container \"e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268\": container with ID starting with e751975a778c2b6fb660beabdbe87b9d8c1f13606db70f54fd669b7081bf5268 not found: ID does not exist"
Nov 02 17:35:07.038376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.038337    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"]
Nov 02 17:35:07.043595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.043568    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-135f915e-74d6cdfdf5-tggmv"]
Nov 02 17:35:07.151508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.151429    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"]
Nov 02 17:35:07.155345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.155322    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-135f915e-6ffdd47fc-kx5fm"]
Nov 02 17:35:07.381848 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.381814    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cd9b55-5f53-460a-996d-c0f9200cae88" path="/var/lib/kubelet/pods/89cd9b55-5f53-460a-996d-c0f9200cae88/volumes"
Nov 02 17:35:07.382274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.382253    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981eff2e-eca9-4423-bb5e-0f43a3122ec6" path="/var/lib/kubelet/pods/981eff2e-eca9-4423-bb5e-0f43a3122ec6/volumes"
Nov 02 17:35:07.485133 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.485052    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:07.714072 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.714035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:07.818568 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:07.818494    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 17:35:08.463078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:08.463046    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:08.784706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:08.784601    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:10.230052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:10.230021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:10.494936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:10.494830    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"]
Nov 02 17:35:10.495254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:10.495195    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy" containerID="cri-o://3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133" gracePeriod=30
Nov 02 17:35:10.514446 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:10.514413    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"]
Nov 02 17:35:10.514714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:10.514672    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6" podUID="d6a57c34-e0f6-4ca4-877f-012a21f4535e" containerName="istio-proxy" containerID="cri-o://0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7" gracePeriod=30
Nov 02 17:35:10.558942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:10.558906    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:11.328671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:11.328641    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:11.774060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:11.774015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:12.061239 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:12.061133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:12.307241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:12.307204    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:15.359681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.359642    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"]
Nov 02 17:35:15.360244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.360097    4476 cpu_manager.go:414] "containerMap: removing container" podUID="89cd9b55-5f53-460a-996d-c0f9200cae88" containerName="payload-pre-processing"
Nov 02 17:35:15.360244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.360111    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cd9b55-5f53-460a-996d-c0f9200cae88" containerName="payload-pre-processing"
Nov 02 17:35:15.360244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.360125    4476 cpu_manager.go:414] "containerMap: removing container" podUID="981eff2e-eca9-4423-bb5e-0f43a3122ec6" containerName="payload-processing"
Nov 02 17:35:15.360244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.360130    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981eff2e-eca9-4423-bb5e-0f43a3122ec6" containerName="payload-processing"
Nov 02 17:35:15.360244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.360242    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="981eff2e-eca9-4423-bb5e-0f43a3122ec6" containerName="payload-processing"
Nov 02 17:35:15.360448 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.360258    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cd9b55-5f53-460a-996d-c0f9200cae88" containerName="payload-pre-processing"
Nov 02 17:35:15.364794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.364773    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.389900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.389867    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.389900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.389897    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cz9\" (UniqueName: \"kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.390212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.390189    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.390340 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.390222    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.417800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.417764    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-6e3175-serving-cert\""
Nov 02 17:35:15.469974 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.469941    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"]
Nov 02 17:35:15.491501 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.491463    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.491501 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.491503    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.491749 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.491566    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.491749 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.491583    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cz9\" (UniqueName: \"kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.491749 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:15.491641    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-serving-cert: secret "maas-api-e2e-worker-w3-6e3175-serving-cert" not found
Nov 02 17:35:15.491749 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:15.491722    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls podName:13a1e239-aa47-4ff3-be58-6dba25de415a nodeName:}" failed. No retries permitted until 2026-11-02 17:35:15.991699944 +0000 UTC m=+746.832305877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls") pod "maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" (UID: "13a1e239-aa47-4ff3-be58-6dba25de415a") : secret "maas-api-e2e-worker-w3-6e3175-serving-cert" not found
Nov 02 17:35:15.492257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.492238    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.494757 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.494736    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.551247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.551213    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cz9\" (UniqueName: \"kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.602573 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.602536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:15.793227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.793199    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:35:15.794202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.794159    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:15.811452 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.811428    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:15.895885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.895787    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.895885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.895828    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.895885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.895868    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.895897    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.895937    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.895989    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896020    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896046    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896076    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896082    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.896189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896109    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896212    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896230    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896333    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896386    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896421    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896452    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.896580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896518    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.897279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896600    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.897279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896675    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data\") pod \"29af9182-405e-455d-93dd-69bff5b0e13b\" (UID: \"29af9182-405e-455d-93dd-69bff5b0e13b\") "
Nov 02 17:35:15.897279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896754    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c\") pod \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\" (UID: \"d6a57c34-e0f6-4ca4-877f-012a21f4535e\") "
Nov 02 17:35:15.897461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.897303    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.897461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.897328    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.897461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.897344    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.897461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.897359    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896423    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896442    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.896808    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.898543    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.898792    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.898940    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:35:15.899049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.899017    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.903287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.899476    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.906039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.905935    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.906264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.906231    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:35:15.906457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.906437    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8" pod "29af9182-405e-455d-93dd-69bff5b0e13b" (UID: "29af9182-405e-455d-93dd-69bff5b0e13b"). InnerVolumeSpecName "kube-api-access-sg8s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.906551 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.906433    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "kube-api-access-gvw5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.906748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.906722    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:15.907082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.906992    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy" pod "d6a57c34-e0f6-4ca4-877f-012a21f4535e" (UID: "d6a57c34-e0f6-4ca4-877f-012a21f4535e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:35:15.935023 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.934987    4476 generic.go:356] "Generic (PLEG): container finished" podID="d6a57c34-e0f6-4ca4-877f-012a21f4535e" containerID="0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7" exitCode=0
Nov 02 17:35:15.935200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.935049    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"
Nov 02 17:35:15.935200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.935066    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6" event={"ID":"d6a57c34-e0f6-4ca4-877f-012a21f4535e","Type":"ContainerDied","Data":"0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7"}
Nov 02 17:35:15.935200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.935153    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6" event={"ID":"d6a57c34-e0f6-4ca4-877f-012a21f4535e","Type":"ContainerDied","Data":"a27a5bc44d0d72869bbcbb19a456a79c0fe56d90283e1d5f70a3c86292506633"}
Nov 02 17:35:15.935200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.935166    4476 scope.go:122] "RemoveContainer" containerID="0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7"
Nov 02 17:35:15.936720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.936695    4476 generic.go:356] "Generic (PLEG): container finished" podID="29af9182-405e-455d-93dd-69bff5b0e13b" containerID="3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133" exitCode=0
Nov 02 17:35:15.936851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.936724    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" event={"ID":"29af9182-405e-455d-93dd-69bff5b0e13b","Type":"ContainerDied","Data":"3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133"}
Nov 02 17:35:15.936851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.936759    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"
Nov 02 17:35:15.936851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.936783    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r" event={"ID":"29af9182-405e-455d-93dd-69bff5b0e13b","Type":"ContainerDied","Data":"39d948bca2883fbdc9e822a8623c02ed78a84d49a732586a1ba523a16163ea98"}
Nov 02 17:35:15.950868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.950464    4476 scope.go:122] "RemoveContainer" containerID="0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7"
Nov 02 17:35:15.950959 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:15.950892    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7\": container with ID starting with 0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7 not found: ID does not exist" containerID="0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7"
Nov 02 17:35:15.950959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.950930    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7"} err="failed to get container status \"0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7\": rpc error: code = NotFound desc = could not find container \"0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7\": container with ID starting with 0f001dac930130f444007af18e3c53dc3deedfc3d5d3cecbe5ae73312fd38ee7 not found: ID does not exist"
Nov 02 17:35:15.950959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.950955    4476 scope.go:122] "RemoveContainer" containerID="3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133"
Nov 02 17:35:15.960881 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.960861    4476 scope.go:122] "RemoveContainer" containerID="3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133"
Nov 02 17:35:15.961213 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:15.961193    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133\": container with ID starting with 3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133 not found: ID does not exist" containerID="3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133"
Nov 02 17:35:15.961268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.961223    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133"} err="failed to get container status \"3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133\": rpc error: code = NotFound desc = could not find container \"3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133\": container with ID starting with 3f41c6773b11b85d5a17c4990b8d1bd5fc7d06ad7acfa0e750d32fe2773da133 not found: ID does not exist"
Nov 02 17:35:15.997986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.997948    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:15.998143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998088    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998105    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af9182-405e-455d-93dd-69bff5b0e13b-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998121    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af9182-405e-455d-93dd-69bff5b0e13b-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998135    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998146    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998160    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998172    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998184    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998196    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998209    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg8s8\" (UniqueName: \"kubernetes.io/projected/29af9182-405e-455d-93dd-69bff5b0e13b-kube-api-access-sg8s8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998224    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998237    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a57c34-e0f6-4ca4-877f-012a21f4535e-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998248    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af9182-405e-455d-93dd-69bff5b0e13b-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:15.998275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:15.998261    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvw5c\" (UniqueName: \"kubernetes.io/projected/d6a57c34-e0f6-4ca4-877f-012a21f4535e-kube-api-access-gvw5c\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:16.000552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.000532    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:16.063663 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.063613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:16.151315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.151223    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"]
Nov 02 17:35:16.157809 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.157781    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t262r"]
Nov 02 17:35:16.268936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.268905    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"]
Nov 02 17:35:16.272873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.272842    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-vpmx6"]
Nov 02 17:35:16.277215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.277195    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:16.516810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.516776    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:16.664105 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.664069    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"]
Nov 02 17:35:16.664730 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:16.664705    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a1e239_aa47_4ff3_be58_6dba25de415a.slice/crio-365758f45e99d055e39cf824c990b89ba902b759f6daf9c7c502aef63315fb82 WatchSource:0}: Error finding container 365758f45e99d055e39cf824c990b89ba902b759f6daf9c7c502aef63315fb82: Status 404 returned error can't find the container with id 365758f45e99d055e39cf824c990b89ba902b759f6daf9c7c502aef63315fb82
Nov 02 17:35:16.708790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.708763    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:16.919894 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.919862    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:16.946028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:16.945993    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" event={"ID":"13a1e239-aa47-4ff3-be58-6dba25de415a","Type":"ContainerStarted","Data":"365758f45e99d055e39cf824c990b89ba902b759f6daf9c7c502aef63315fb82"}
Nov 02 17:35:17.140065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:17.140028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:17.362060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:17.362024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:17.382223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:17.382187    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" path="/var/lib/kubelet/pods/29af9182-405e-455d-93dd-69bff5b0e13b/volumes"
Nov 02 17:35:17.382600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:17.382587    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a57c34-e0f6-4ca4-877f-012a21f4535e" path="/var/lib/kubelet/pods/d6a57c34-e0f6-4ca4-877f-012a21f4535e/volumes"
Nov 02 17:35:17.954419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:17.954384    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" event={"ID":"13a1e239-aa47-4ff3-be58-6dba25de415a","Type":"ContainerStarted","Data":"df6134b879d78a7651ef4cb352abca6094bf54c6f231e9c026b5f32f09ad50aa"}
Nov 02 17:35:17.954846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:17.954443    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:18.071880 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:18.071826    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" podStartSLOduration=2.218499165 podStartE2EDuration="3.071812725s" totalImagesPullingTime="853.31356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:18.069401694 +0000 UTC m=+748.910007629" watchObservedRunningTime="2026-11-02 17:35:18.071812725 +0000 UTC m=+748.912418659"
Nov 02 17:35:20.428574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.428540    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"]
Nov 02 17:35:20.429047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.428764    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" podUID="87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" containerName="maas-api" containerID="cri-o://56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2" gracePeriod=30
Nov 02 17:35:20.583234 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.583191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:20.694105 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.694080    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:35:20.742705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.742667    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\") pod \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") "
Nov 02 17:35:20.742886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.742715    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca\") pod \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") "
Nov 02 17:35:20.742886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.742809    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs\") pod \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") "
Nov 02 17:35:20.742886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.742883    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf\" (UniqueName: \"kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf\") pod \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\" (UID: \"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e\") "
Nov 02 17:35:20.743101 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.743077    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca" pod "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" (UID: "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:35:20.743208 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.743189    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:20.745081 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.745053    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs" pod "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" (UID: "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:35:20.745179 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.745096    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls" pod "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" (UID: "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:35:20.745179 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.745113    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf" pod "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" (UID: "87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e"). InnerVolumeSpecName "kube-api-access-nxgvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:20.844658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.844599    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxgvf\" (UniqueName: \"kubernetes.io/projected/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-kube-api-access-nxgvf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:20.844658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.844661    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:20.844861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.844676    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:35:20.978018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.977924    4476 generic.go:356] "Generic (PLEG): container finished" podID="87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" containerID="56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2" exitCode=0
Nov 02 17:35:20.978156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.978041    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" event={"ID":"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e","Type":"ContainerDied","Data":"56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2"}
Nov 02 17:35:20.978156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.978086    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p" event={"ID":"87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e","Type":"ContainerDied","Data":"8070d20ee969c158a52e1fcda58710d19b8f767f22523178f4f1c61e524af36a"}
Nov 02 17:35:20.978156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.978109    4476 scope.go:122] "RemoveContainer" containerID="56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2"
Nov 02 17:35:20.978273 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.978233    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"
Nov 02 17:35:20.988790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.988768    4476 scope.go:122] "RemoveContainer" containerID="56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2"
Nov 02 17:35:20.989077 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:20.989057    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2\": container with ID starting with 56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2 not found: ID does not exist" containerID="56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2"
Nov 02 17:35:20.989135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:20.989086    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2"} err="failed to get container status \"56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2\": rpc error: code = NotFound desc = could not find container \"56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2\": container with ID starting with 56b1717a3fcab7d6a91b51a1f8a3ffcb8255f03c5524d6886b99018693c42cc2 not found: ID does not exist"
Nov 02 17:35:21.142661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:21.142613    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"]
Nov 02 17:35:21.146839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:21.146817    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-135f915e-bbbc65fb-dsx9p"]
Nov 02 17:35:21.381686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:21.381657    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" path="/var/lib/kubelet/pods/87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e/volumes"
Nov 02 17:35:22.239254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:22.239217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:22.473039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:22.472999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:22.729495 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:22.729465    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:22.923227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:22.923194    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:23.761128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:23.761092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:23.967120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:23.967071    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 17:35:23.977665 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:23.975452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:24.184806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.184759    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:24.289205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289164    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"]
Nov 02 17:35:24.289789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289768    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d6a57c34-e0f6-4ca4-877f-012a21f4535e" containerName="istio-proxy"
Nov 02 17:35:24.289789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289790    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a57c34-e0f6-4ca4-877f-012a21f4535e" containerName="istio-proxy"
Nov 02 17:35:24.289973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289805    4476 cpu_manager.go:414] "containerMap: removing container" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy"
Nov 02 17:35:24.289973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289812    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy"
Nov 02 17:35:24.289973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289846    4476 cpu_manager.go:414] "containerMap: removing container" podUID="87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" containerName="maas-api"
Nov 02 17:35:24.289973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289853    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" containerName="maas-api"
Nov 02 17:35:24.289973 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289964    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="87df5c6b-f6a0-42e6-a58b-f1ef1c053d7e" containerName="maas-api"
Nov 02 17:35:24.290223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289978    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a57c34-e0f6-4ca4-877f-012a21f4535e" containerName="istio-proxy"
Nov 02 17:35:24.290223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.289990    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="29af9182-405e-455d-93dd-69bff5b0e13b" containerName="istio-proxy"
Nov 02 17:35:24.294506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.294483    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.360835 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.360799    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4ad655\""
Nov 02 17:35:24.367266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.367235    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4ad655-dockercfg-7wxrk\""
Nov 02 17:35:24.377887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.377852    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"]
Nov 02 17:35:24.379081 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.379059    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.379157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.379113    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsjp\" (UniqueName: \"kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp\") pod \"payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.423184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.423153    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"]
Nov 02 17:35:24.426755 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.426735    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.480455 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.480358    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.480455 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.480419    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncv6\" (UniqueName: \"kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6\") pod \"payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.480710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.480461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.480710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.480560    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsjp\" (UniqueName: \"kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp\") pod \"payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.481184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.481165    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.506189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.506138    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:24.506453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.506225    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"]
Nov 02 17:35:24.555019 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.554983    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsjp\" (UniqueName: \"kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp\") pod \"payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.581750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.581715    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncv6\" (UniqueName: \"kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6\") pod \"payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.581945 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.581764    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.582461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.582440    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.621340 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.621311    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:24.638298 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.638271    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncv6\" (UniqueName: \"kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6\") pod \"payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:24.707252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.707224    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:24.739786 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:24.739702    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:25.007222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:25.007199    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"]
Nov 02 17:35:25.009332 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:25.009305    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcae5a9_0fdc_4c39_8025_e5f714fbf883.slice/crio-55ad3543276464afa9205547ed916c220f62eae1eadbb2d2571f078c0f55ae02 WatchSource:0}: Error finding container 55ad3543276464afa9205547ed916c220f62eae1eadbb2d2571f078c0f55ae02: Status 404 returned error can't find the container with id 55ad3543276464afa9205547ed916c220f62eae1eadbb2d2571f078c0f55ae02
Nov 02 17:35:25.127143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:25.127117    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"]
Nov 02 17:35:25.129185 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:25.129163    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54391c3_dd1f_4e0f_b407_e91611f79e93.slice/crio-2e921309ff853685c921d4538f641f89b131cb348ee320ca14aee4141ad892a7 WatchSource:0}: Error finding container 2e921309ff853685c921d4538f641f89b131cb348ee320ca14aee4141ad892a7: Status 404 returned error can't find the container with id 2e921309ff853685c921d4538f641f89b131cb348ee320ca14aee4141ad892a7
Nov 02 17:35:26.014304 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.014267    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz" event={"ID":"f54391c3-dd1f-4e0f-b407-e91611f79e93","Type":"ContainerStarted","Data":"41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d"}
Nov 02 17:35:26.014774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.014318    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz" event={"ID":"f54391c3-dd1f-4e0f-b407-e91611f79e93","Type":"ContainerStarted","Data":"2e921309ff853685c921d4538f641f89b131cb348ee320ca14aee4141ad892a7"}
Nov 02 17:35:26.014774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.014390    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:26.016202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.016176    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm" event={"ID":"3fcae5a9-0fdc-4c39-8025-e5f714fbf883","Type":"ContainerStarted","Data":"572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8"}
Nov 02 17:35:26.016321 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.016221    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm" event={"ID":"3fcae5a9-0fdc-4c39-8025-e5f714fbf883","Type":"ContainerStarted","Data":"55ad3543276464afa9205547ed916c220f62eae1eadbb2d2571f078c0f55ae02"}
Nov 02 17:35:26.016321 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.016309    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:26.131409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.131356    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz" podStartSLOduration=2.131342187 podStartE2EDuration="2.131342187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:26.129463107 +0000 UTC m=+756.970069045" watchObservedRunningTime="2026-11-02 17:35:26.131342187 +0000 UTC m=+756.971948121"
Nov 02 17:35:26.253604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:26.253546    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm" podStartSLOduration=2.253529688 podStartE2EDuration="2.253529688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:26.252406296 +0000 UTC m=+757.093012230" watchObservedRunningTime="2026-11-02 17:35:26.253529688 +0000 UTC m=+757.094135626"
Nov 02 17:35:27.661014 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:27.660975    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:27.858170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:27.858140    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:28.048689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:28.048589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:29.057696 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:29.057663    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:35:30.245136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:30.245095    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:30.514190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:30.514092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:32.378649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:32.378607    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:33.007872 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:33.007835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:33.227515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:33.227482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:33.428023 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:33.427985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:36.894013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:36.893970    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:37.025342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:37.025310    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 17:35:37.025552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:37.025399    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 17:35:38.435022 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.434985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:38.723280 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.723185    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:38.832313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.832277    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"]
Nov 02 17:35:38.836194 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.836163    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:38.888734 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.888702    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dc5689\""
Nov 02 17:35:38.909323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.909291    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:38.909536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.909331    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqxc\" (UniqueName: \"kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc\") pod \"payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:38.938291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.938259    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dc5689-dockercfg-vxxrs\""
Nov 02 17:35:38.940992 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.940966    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"]
Nov 02 17:35:38.943961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.943930    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"]
Nov 02 17:35:38.947460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.947441    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:38.960128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.960099    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:38.999925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:38.999839    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"]
Nov 02 17:35:39.010280 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.010247    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2pg\" (UniqueName: \"kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg\") pod \"payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.010476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.010300    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:39.010476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.010376    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.010586 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.010492    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqxc\" (UniqueName: \"kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc\") pod \"payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:39.011287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.011260    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:39.068470 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.068434    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqxc\" (UniqueName: \"kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc\") pod \"payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:39.111672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.111613    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2pg\" (UniqueName: \"kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg\") pod \"payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.111854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.111701    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.112546 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.112520    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.150602 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.150571    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:39.170198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.170166    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2pg\" (UniqueName: \"kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg\") pod \"payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.260661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.260548    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:39.542005 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.541978    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"]
Nov 02 17:35:39.543465 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:39.543436    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc33efd_3688_4ef5_8298_00f8d0e7554b.slice/crio-94056df4ce88c38e6c113f92a1f79a2879ec0ca835492e5d2bb611151b5e0194 WatchSource:0}: Error finding container 94056df4ce88c38e6c113f92a1f79a2879ec0ca835492e5d2bb611151b5e0194: Status 404 returned error can't find the container with id 94056df4ce88c38e6c113f92a1f79a2879ec0ca835492e5d2bb611151b5e0194
Nov 02 17:35:39.651241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:39.651218    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"]
Nov 02 17:35:39.653786 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:39.653757    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c22053b_aceb_4f84_a364_78c65b651ac8.slice/crio-ea3357d63bfd359b845ce7d8534e00e842ee07dd203edf721b90b848fa591630 WatchSource:0}: Error finding container ea3357d63bfd359b845ce7d8534e00e842ee07dd203edf721b90b848fa591630: Status 404 returned error can't find the container with id ea3357d63bfd359b845ce7d8534e00e842ee07dd203edf721b90b848fa591630
Nov 02 17:35:40.102534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.102495    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" event={"ID":"9c22053b-aceb-4f84-a364-78c65b651ac8","Type":"ContainerStarted","Data":"3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff"}
Nov 02 17:35:40.102767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.102680    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:40.102767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.102722    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" event={"ID":"9c22053b-aceb-4f84-a364-78c65b651ac8","Type":"ContainerStarted","Data":"ea3357d63bfd359b845ce7d8534e00e842ee07dd203edf721b90b848fa591630"}
Nov 02 17:35:40.104101 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.104076    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" event={"ID":"6fc33efd-3688-4ef5-8298-00f8d0e7554b","Type":"ContainerStarted","Data":"6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347"}
Nov 02 17:35:40.104209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.104115    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" event={"ID":"6fc33efd-3688-4ef5-8298-00f8d0e7554b","Type":"ContainerStarted","Data":"94056df4ce88c38e6c113f92a1f79a2879ec0ca835492e5d2bb611151b5e0194"}
Nov 02 17:35:40.104209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.104140    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:40.224683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.221738    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" podStartSLOduration=2.221719753 podStartE2EDuration="2.221719753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:40.21816153 +0000 UTC m=+771.058767465" watchObservedRunningTime="2026-11-02 17:35:40.221719753 +0000 UTC m=+771.062325691"
Nov 02 17:35:40.333307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:40.333237    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" podStartSLOduration=2.333219858 podStartE2EDuration="2.333219858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:40.332001192 +0000 UTC m=+771.172607126" watchObservedRunningTime="2026-11-02 17:35:40.333219858 +0000 UTC m=+771.173825793"
Nov 02 17:35:43.931673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:43.931608    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:44.146024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:44.145985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:49.382962 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:49.382924    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:50.393306 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:50.393276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:51.042682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:51.042638    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:51.113361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:51.113333    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 17:35:51.113557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:51.113503    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 17:35:51.324195 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:51.324118    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:54.532768 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.532733    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"]
Nov 02 17:35:54.537940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.537920    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.642065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.642036    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-dc5689-serving-cert\""
Nov 02 17:35:54.644622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.644600    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"]
Nov 02 17:35:54.662258 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.662219    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.662423 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.662279    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.662423 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.662392    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.662585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.662426    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jcr\" (UniqueName: \"kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.763261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.763224    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.763261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.763264    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.763510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.763294    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.763510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.763316    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jcr\" (UniqueName: \"kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.763510 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:54.763395    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-serving-cert: secret "maas-api-e2e-worker-w1-dc5689-serving-cert" not found
Nov 02 17:35:54.763510 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:54.763491    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls podName:7016d668-cd4b-4860-9908-2a97e39167a1 nodeName:}" failed. No retries permitted until 2026-11-02 17:35:55.263469362 +0000 UTC m=+786.104075276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls") pod "maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" (UID: "7016d668-cd4b-4860-9908-2a97e39167a1") : secret "maas-api-e2e-worker-w1-dc5689-serving-cert" not found
Nov 02 17:35:54.764033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.764014    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.765994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.765976    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.778908 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.778877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:54.820209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.820139    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jcr\" (UniqueName: \"kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:54.966135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:54.966097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:55.159916 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:55.159882    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:55.268352 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:55.268305    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:55.271120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:55.271089    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:55.450575 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:55.450486    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:55.831803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:55.831776    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"]
Nov 02 17:35:55.834068 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:35:55.834035    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7016d668_cd4b_4860_9908_2a97e39167a1.slice/crio-246d832ca321962011796cf88ea6067362faf07016ffbe0e9cf68ad807498d15 WatchSource:0}: Error finding container 246d832ca321962011796cf88ea6067362faf07016ffbe0e9cf68ad807498d15: Status 404 returned error can't find the container with id 246d832ca321962011796cf88ea6067362faf07016ffbe0e9cf68ad807498d15
Nov 02 17:35:56.208124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:56.208090    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" event={"ID":"7016d668-cd4b-4860-9908-2a97e39167a1","Type":"ContainerStarted","Data":"246d832ca321962011796cf88ea6067362faf07016ffbe0e9cf68ad807498d15"}
Nov 02 17:35:57.217331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:57.217297    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" event={"ID":"7016d668-cd4b-4860-9908-2a97e39167a1","Type":"ContainerStarted","Data":"2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02"}
Nov 02 17:35:57.217739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:57.217501    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:35:57.332577 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:57.332524    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" podStartSLOduration=2.417823818 podStartE2EDuration="3.332508384s" totalImagesPullingTime="914.684566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:57.332307399 +0000 UTC m=+788.172913335" watchObservedRunningTime="2026-11-02 17:35:57.332508384 +0000 UTC m=+788.173114317"
Nov 02 17:35:59.059585 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:35:59.059545    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:35:59.880413 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:35:59.880377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:00.043795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.043760    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"]
Nov 02 17:36:00.048223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.048195    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.153591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.153483    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"kube-root-ca.crt\""
Nov 02 17:36:00.154292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.153677    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"default-dockercfg-fvn5n\""
Nov 02 17:36:00.154292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.153726    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"openshift-service-ca.crt\""
Nov 02 17:36:00.154292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.153803    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"simulator-w3-6e3175-kserve-self-signed-certs\""
Nov 02 17:36:00.157168 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.157141    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"]
Nov 02 17:36:00.160705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.160674    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:00.211721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.211681    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.211886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.211725    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj57d\" (UniqueName: \"kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.211886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.211829    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.212002 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.211893    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.212051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.212025    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.212107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.212094    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313286    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313344    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj57d\" (UniqueName: \"kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313388    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313427    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313489    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313779 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313571    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.313779 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.313764    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.314038 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.314009    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.314524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.314487    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.316643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.316576    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.316643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.316602    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.371712 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.371676    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj57d\" (UniqueName: \"kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d\") pod \"simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") " pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:00.660701 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:00.660663    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:01.035969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:01.035933    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"]
Nov 02 17:36:01.038364 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:36:01.038336    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7a8d9e2_17ee_4b30_9240_3c33043cc156.slice/crio-0db6312f203910f4f4db27975346a7d178dc53cef209b5fd890d067900a3b7d4 WatchSource:0}: Error finding container 0db6312f203910f4f4db27975346a7d178dc53cef209b5fd890d067900a3b7d4: Status 404 returned error can't find the container with id 0db6312f203910f4f4db27975346a7d178dc53cef209b5fd890d067900a3b7d4
Nov 02 17:36:01.243764 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:01.243720    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" event={"ID":"f7a8d9e2-17ee-4b30-9240-3c33043cc156","Type":"ContainerStarted","Data":"0db6312f203910f4f4db27975346a7d178dc53cef209b5fd890d067900a3b7d4"}
Nov 02 17:36:03.230988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:03.230962    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 17:36:04.267138 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:04.267092    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" event={"ID":"f7a8d9e2-17ee-4b30-9240-3c33043cc156","Type":"ContainerStarted","Data":"180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688"}
Nov 02 17:36:04.381644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:04.381584    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" podStartSLOduration=3.197700656 podStartE2EDuration="5.381569875s" totalImagesPullingTime="2.183869219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:04.380252113 +0000 UTC m=+795.220858046" watchObservedRunningTime="2026-11-02 17:36:04.381569875 +0000 UTC m=+795.222175809"
Nov 02 17:36:04.661509 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:04.661454    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:04.822868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:04.822838    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:07.407424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:07.407379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:08.015411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:08.015338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:08.240514 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:08.240463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:10.217087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:10.217054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:10.661198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:10.661169    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:10.674228 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:10.674203    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:11.322611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:11.322581    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:36:14.963846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:14.963808    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:16.646165 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:16.646124    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:17.274225 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:17.274192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:17.503032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:17.503000    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:20.105798 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:20.105766    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:24.204591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:24.204556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:24.897746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:24.897701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:29.059249 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:36:29.059204    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:36:29.678968 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:29.678931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:29.964940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:29.961869    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:34.682187 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:34.682158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:34.938330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:34.938242    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:35.564622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:35.564585    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:36.809160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:36.809126    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:39.568981 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.568942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:39.587892 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.587858    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"]
Nov 02 17:36:39.591362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.591344    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.644079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.644048    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"openshift-service-ca.crt\""
Nov 02 17:36:39.644079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.644067    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"kube-root-ca.crt\""
Nov 02 17:36:39.644567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.644552    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"default-dockercfg-tm5n2\""
Nov 02 17:36:39.675548 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.675522    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.675709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.675558    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9sgz\" (UniqueName: \"kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.675709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.675598    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.675813 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.675710    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.675813 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.675788    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.675896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.675818    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.693886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.693857    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"simulator-w4-4ad655-kserve-self-signed-certs\""
Nov 02 17:36:39.696623 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.696600    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"]
Nov 02 17:36:39.769123 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.769095    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:39.776457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.776430    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.776605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.776464    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.776605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.776496    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.776605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.776530    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.776859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.776605    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.776859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.776650    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9sgz\" (UniqueName: \"kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.777234 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.777215    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.777668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.777612    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.777902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.777880    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.779813 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.779786    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.780189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.780171    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.835267 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.835193    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9sgz\" (UniqueName: \"kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz\") pod \"simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") " pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:39.907254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:39.907212    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:40.301037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:40.301012    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"]
Nov 02 17:36:40.303189 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:36:40.303158    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef20ad96_c209_4313_8a19_02c3d9b5d999.slice/crio-29987ca1f6ddd223640647163a78ebbcf1342f361bdeed387cbf942134430e2c WatchSource:0}: Error finding container 29987ca1f6ddd223640647163a78ebbcf1342f361bdeed387cbf942134430e2c: Status 404 returned error can't find the container with id 29987ca1f6ddd223640647163a78ebbcf1342f361bdeed387cbf942134430e2c
Nov 02 17:36:40.480011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:40.479969    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" event={"ID":"ef20ad96-c209-4313-8a19-02c3d9b5d999","Type":"ContainerStarted","Data":"e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a"}
Nov 02 17:36:40.480011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:40.480016    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" event={"ID":"ef20ad96-c209-4313-8a19-02c3d9b5d999","Type":"ContainerStarted","Data":"29987ca1f6ddd223640647163a78ebbcf1342f361bdeed387cbf942134430e2c"}
Nov 02 17:36:40.598790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:40.598674    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" podStartSLOduration=1.5986573499999999 podStartE2EDuration="1.59865735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:40.596097196 +0000 UTC m=+831.436703131" watchObservedRunningTime="2026-11-02 17:36:40.59865735 +0000 UTC m=+831.439263283"
Nov 02 17:36:40.907865 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:40.907830    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:44.245336 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:44.245300    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:44.430545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:44.430506    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:48.952380 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:48.952336    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:49.908545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:49.908510    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:49.921364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:49.921330    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:50.552086 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:50.552060    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 17:36:53.657478 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:53.657433    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:58.459938 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.459907    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"]
Nov 02 17:36:58.462643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.462597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:58.462981 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.462964    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.534826 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.534797    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"]
Nov 02 17:36:58.562746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.562712    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.562894 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.562749    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.563034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.563017    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqwdx\" (UniqueName: \"kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.563098 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.563084    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.563154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.563124    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.563154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.563144    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.563871 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.563847    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"premium-w3-6e3175-kserve-self-signed-certs\""
Nov 02 17:36:58.656988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.656951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:58.664203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664167    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.664390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664221    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.664390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664319    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqwdx\" (UniqueName: \"kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.664390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664364    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.664561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664416    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.664561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664438    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.664964 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.664941    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.665063 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.665042    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.665328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.665296    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.667574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.667550    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.667821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.667800    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.721782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.721698    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqwdx\" (UniqueName: \"kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx\") pod \"premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") " pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:58.799214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:58.799174    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:36:59.059473 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:36:59.059391    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:36:59.184118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:59.184096    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"]
Nov 02 17:36:59.185734 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:36:59.185708    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb1731d5_2413_48d5_932a_cf777619c149.slice/crio-8b7ea16310969345bf94efb373e914b57ee096682cc74a3799e103def20aadae WatchSource:0}: Error finding container 8b7ea16310969345bf94efb373e914b57ee096682cc74a3799e103def20aadae: Status 404 returned error can't find the container with id 8b7ea16310969345bf94efb373e914b57ee096682cc74a3799e103def20aadae
Nov 02 17:36:59.600654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:59.600594    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" event={"ID":"eb1731d5-2413-48d5-932a-cf777619c149","Type":"ContainerStarted","Data":"5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe"}
Nov 02 17:36:59.601112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:59.600666    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" event={"ID":"eb1731d5-2413-48d5-932a-cf777619c149","Type":"ContainerStarted","Data":"8b7ea16310969345bf94efb373e914b57ee096682cc74a3799e103def20aadae"}
Nov 02 17:36:59.716721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:59.716665    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" podStartSLOduration=1.716646433 podStartE2EDuration="1.716646433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:59.71566008 +0000 UTC m=+850.556266011" watchObservedRunningTime="2026-11-02 17:36:59.716646433 +0000 UTC m=+850.557252370"
Nov 02 17:36:59.799600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:36:59.799569    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:37:03.181088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:03.180978    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:06.265213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:06.265038    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:08.008079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:08.008029    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:08.799969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:08.799919    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:37:08.813925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:08.813887    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:37:09.668897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:09.668861    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 17:37:12.767553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:12.767519    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:17.456059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:17.456030    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:22.150437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:22.150355    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:26.862808 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:26.862751    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:29.059087 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:37:29.059045    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:37:31.569388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:31.569348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:32.775244 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:32.775209    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:36.386003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:36.385964    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:36.580838 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:36.580796    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:41.262580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:41.262542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:44.571219 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:44.571181    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:45.967834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:45.967788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:48.548014 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.547972    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"]
Nov 02 17:37:48.552116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.552094    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.651152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.651110    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"]
Nov 02 17:37:48.656746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.656720    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ad655\"/\"premium-w4-4ad655-kserve-self-signed-certs\""
Nov 02 17:37:48.658736 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.658710    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.658875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.658741    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.658875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.658808    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.659004 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.658897    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5vr\" (UniqueName: \"kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.659004 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.658974    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.659113 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.659052    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.685295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.685261    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:48.759732 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.759694    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.759916 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.759737    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.759916 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.759791    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.759916 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.759813    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.759916 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.759887    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.760139 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.759926    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5vr\" (UniqueName: \"kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.760479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.760449    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.760650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.760583    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.760765 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.760743    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.762519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.762487    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.763314 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.763283    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.816435 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.816338    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5vr\" (UniqueName: \"kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr\") pod \"premium-w4-4ad655-kserve-769756558c-7wr78\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") " pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:48.867504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:48.867460    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:49.275025 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:49.274998    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"]
Nov 02 17:37:49.277247 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:37:49.277221    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f24edb2_2853_4d83_ad30_4d4d64d7823c.slice/crio-86b74ee5a14193788ebd3df397029065d4b3095c8f7077dccd1f5ba54ff8847d WatchSource:0}: Error finding container 86b74ee5a14193788ebd3df397029065d4b3095c8f7077dccd1f5ba54ff8847d: Status 404 returned error can't find the container with id 86b74ee5a14193788ebd3df397029065d4b3095c8f7077dccd1f5ba54ff8847d
Nov 02 17:37:49.468518 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:49.468490    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:37:49.474377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:49.474348    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:37:49.923694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:49.923653    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" event={"ID":"5f24edb2-2853-4d83-ad30-4d4d64d7823c","Type":"ContainerStarted","Data":"fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410"}
Nov 02 17:37:49.923694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:49.923700    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" event={"ID":"5f24edb2-2853-4d83-ad30-4d4d64d7823c","Type":"ContainerStarted","Data":"86b74ee5a14193788ebd3df397029065d4b3095c8f7077dccd1f5ba54ff8847d"}
Nov 02 17:37:50.040673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:50.040601    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" podStartSLOduration=2.040585595 podStartE2EDuration="2.040585595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:50.038018648 +0000 UTC m=+900.878624612" watchObservedRunningTime="2026-11-02 17:37:50.040585595 +0000 UTC m=+900.881191528"
Nov 02 17:37:50.685517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:50.685484    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:50.868586 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:50.868547    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:55.371607 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:55.371562    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:58.868117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:58.868079    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:58.881039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:58.881012    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:58.989041 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:37:58.989014    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 17:37:59.059774 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:37:59.059732    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:38:00.146898 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:00.146846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:04.866651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:04.866449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:09.568509 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:09.568461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:11.313300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:11.313258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:14.281418 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:14.281385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:19.021146 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:19.021108    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:23.786595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:23.786483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:28.589353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:28.589304    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:29.058496 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:38:29.058453    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:38:33.374295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:33.374211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:35.157178 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:35.157117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:35.360956 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:35.360922    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:38.082419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:38.082376    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:41.699672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:41.699619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:42.766056 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:42.766001    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:44.924589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:44.924553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:47.557859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:47.557815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:52.276173 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:52.276134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:56.971278 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:56.971241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:57.941844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:57.941805    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:58.139493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.139452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:58.326852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.326767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:58.734688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.734656    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"]
Nov 02 17:38:58.738353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.738334    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.791274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.791250    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ad655-openshift-default-dockercfg-jbjjg\""
Nov 02 17:38:58.843696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.843661    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"]
Nov 02 17:38:58.867040 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867013    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867066    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867127    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867149    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867178    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867345    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867384    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867441    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.867496 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.867489    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4wt\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968102 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968068    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968102 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968101    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968117    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968139    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968172    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968210    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968239    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968272    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4wt\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968331    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968505    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.968761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.968562    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.969255 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.969232    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.969338 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.969315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.969614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.969594    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.970802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.970774    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:58.970902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:58.970813    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:59.025106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.025036    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4wt\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:59.025255 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.025117    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:59.051554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.051520    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:38:59.068254 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:38:59.068216    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:38:59.486527 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.486503    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"]
Nov 02 17:38:59.488048 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:38:59.488025    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6ad33a_a039_4e1d_acc4_5b6a99583279.slice/crio-eaf4878109f583140642c09e5a717c7a8f63d7ac1b8a860206a0bf7ba7ead447 WatchSource:0}: Error finding container eaf4878109f583140642c09e5a717c7a8f63d7ac1b8a860206a0bf7ba7ead447: Status 404 returned error can't find the container with id eaf4878109f583140642c09e5a717c7a8f63d7ac1b8a860206a0bf7ba7ead447
Nov 02 17:38:59.490331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.490303    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:38:59.490412 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.490362    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:38:59.490412 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:38:59.490407    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:00.373708 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:00.373662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" event={"ID":"4e6ad33a-a039-4e1d-acc4-5b6a99583279","Type":"ContainerStarted","Data":"d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03"}
Nov 02 17:39:00.373927 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:00.373730    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" event={"ID":"4e6ad33a-a039-4e1d-acc4-5b6a99583279","Type":"ContainerStarted","Data":"eaf4878109f583140642c09e5a717c7a8f63d7ac1b8a860206a0bf7ba7ead447"}
Nov 02 17:39:00.490862 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:00.490813    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podStartSLOduration=2.49079931 podStartE2EDuration="2.49079931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:00.488323413 +0000 UTC m=+971.328929348" watchObservedRunningTime="2026-11-02 17:39:00.49079931 +0000 UTC m=+971.331405244"
Nov 02 17:39:01.058654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:01.057975    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:39:01.060382 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:01.060353    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:01.060600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:01.060571    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 17:39:01.579563 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:01.579536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:01.800060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:01.800017    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:02.052982 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:02.052947    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:02.053307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:02.053264    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 17:39:03.054125 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:03.052064    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:03.054125 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:03.052130    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 17:39:04.053039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:04.052998    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:04.053319 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:04.053289    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 17:39:05.056654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:05.052004    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 17:39:05.056654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:05.052068    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 17:39:06.189860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:06.189822    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:39:06.190369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:06.190332    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:39:06.191324 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:06.191305    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 17:39:06.475745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:06.475656    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:06.955732 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:06.955698    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:11.274257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:11.274217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:15.975746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:15.975677    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:17.629753 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:17.629716    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:20.767431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:20.767399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:21.998365 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:21.998328    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"]
Nov 02 17:39:22.002471 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.002450    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108429    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108518    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108594    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108697    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26s9\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108952 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108799    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.108952 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108922    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.109054 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.108954    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.209993    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210055    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210091    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210122    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210158    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210225    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210263    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26s9\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210312    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210386    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210678 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210453    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210678 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210507    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210779    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.210884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.210866    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.211141 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.211117    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.212906 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.212878    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.213158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.213138    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.226114 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.226089    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"]
Nov 02 17:39:22.267467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.267402    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.267999 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.267976    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26s9\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-vm5xs\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.318616 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.318586    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:22.752415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.752389    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"]
Nov 02 17:39:22.754715 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:39:22.754689    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5d2314_e566_4b04_92e8_08c64f93164b.slice/crio-d46c4e5513df31213a29029e3f20dc4eac8d1b0f02e79d404d39d9472d0c1591 WatchSource:0}: Error finding container d46c4e5513df31213a29029e3f20dc4eac8d1b0f02e79d404d39d9472d0c1591: Status 404 returned error can't find the container with id d46c4e5513df31213a29029e3f20dc4eac8d1b0f02e79d404d39d9472d0c1591
Nov 02 17:39:22.756929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.756895    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:22.757012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.756970    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:22.757012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:22.757000    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:23.520266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:23.520219    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" event={"ID":"3d5d2314-e566-4b04-92e8-08c64f93164b","Type":"ContainerStarted","Data":"6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1"}
Nov 02 17:39:23.520266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:23.520270    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" event={"ID":"3d5d2314-e566-4b04-92e8-08c64f93164b","Type":"ContainerStarted","Data":"d46c4e5513df31213a29029e3f20dc4eac8d1b0f02e79d404d39d9472d0c1591"}
Nov 02 17:39:23.688661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:23.688596    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podStartSLOduration=2.688583554 podStartE2EDuration="2.688583554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:23.685446814 +0000 UTC m=+994.526052748" watchObservedRunningTime="2026-11-02 17:39:23.688583554 +0000 UTC m=+994.529189487"
Nov 02 17:39:24.319564 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:24.319524    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:24.321154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:24.321133    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:24.321231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:24.321179    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:25.319173 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:25.319128    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:25.319578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:25.319202    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:25.654656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:25.653150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:26.096574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:26.096488    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:26.319201 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:26.319162    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:26.319597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:26.319218    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:27.319397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:27.319351    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:27.319981 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:27.319411    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:28.319685 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:28.319642    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:28.320090 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:28.319709    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:29.063713 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:39:29.063665    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:39:29.319522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:29.319436    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:29.319810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:29.319780    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:30.319499 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:30.319458    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:39:30.319711 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:30.319523    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 02 17:39:30.556906 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:30.555538    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:31.472614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:31.472562    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:31.473281 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:31.473255    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:31.474039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:31.474012    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 17:39:35.425234 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:35.425188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:37.118519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.118488    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"]
Nov 02 17:39:37.122032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.122016    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195215    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2znr\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195248    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195279    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195385    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195518 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195563 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195547    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195573    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195597    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.195734 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.195720    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296396    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296439    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296466    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296511    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296529    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2znr\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296553    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296593    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296622    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.296720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296687    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.297100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296885    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.297100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.296942    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.297389 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.297350    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.297734 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.297551    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.297857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.297789    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.299112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.299092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.299594 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.299573    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.354335 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.354299    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.354688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.354665    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2znr\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-k4glj\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.363700 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.363672    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"]
Nov 02 17:39:37.435560 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.435480    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:37.867738 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.867706    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"]
Nov 02 17:39:37.908894 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:39:37.908851    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50570585_6b43_4ac9_80a7_df1e86aa8307.slice/crio-c18ddbf24bf8790149dde970d8d2b1c817a44e12afc1bc9da21950221020af98 WatchSource:0}: Error finding container c18ddbf24bf8790149dde970d8d2b1c817a44e12afc1bc9da21950221020af98: Status 404 returned error can't find the container with id c18ddbf24bf8790149dde970d8d2b1c817a44e12afc1bc9da21950221020af98
Nov 02 17:39:37.911325 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.911295    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:37.911410 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.911360    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:37.911410 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:37.911388    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:39:38.662654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:38.661839    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" event={"ID":"50570585-6b43-4ac9-80a7-df1e86aa8307","Type":"ContainerStarted","Data":"624ec44376df0b442875e9c80a22af60377dada97888bd4e8ad6555a1ef859f6"}
Nov 02 17:39:38.662654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:38.661923    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" event={"ID":"50570585-6b43-4ac9-80a7-df1e86aa8307","Type":"ContainerStarted","Data":"c18ddbf24bf8790149dde970d8d2b1c817a44e12afc1bc9da21950221020af98"}
Nov 02 17:39:38.882189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:38.882141    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podStartSLOduration=1.882127762 podStartE2EDuration="1.882127762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:38.831563053 +0000 UTC m=+1009.672168989" watchObservedRunningTime="2026-11-02 17:39:38.882127762 +0000 UTC m=+1009.722733733"
Nov 02 17:39:39.436570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:39.436542    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:39.450539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:39.450505    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:39.450851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:39.450821    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:40.363652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:40.357111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:40.436125 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:40.436097    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:40.436281 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:40.436144    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:41.436210 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:41.436173    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:41.436663 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:41.436244    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:42.436271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:42.436232    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:42.436271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:42.436285    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:42.493128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:42.493098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:43.436135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:43.436099    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:43.436316 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:43.436165    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:44.436724 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:44.436685    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:44.437153 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:44.436743    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:45.275664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:45.275086    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:45.436047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:45.436010    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:39:45.436227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:45.436071    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 17:39:45.566722 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:45.564546    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:45.753194 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:45.753133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:47.437076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:47.437033    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:39:47.437536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:47.437120    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:39:48.478841 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:48.478781    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:39:48.479330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:48.478864    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:39:48.976773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:48.976740    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:48.977377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:48.977145    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:48.978238 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:48.978215    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:39:50.308234 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:50.308193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:55.052806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:55.052770    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:57.237783 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.237749    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"]
Nov 02 17:39:57.242581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.242548    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.342397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.342365    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.342569 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.342404    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv665\" (UniqueName: \"kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.342569 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.342428    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.342689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.342588    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.342689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.342610    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.342774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.342724    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.343832 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.343810    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"kube-root-ca.crt\""
Nov 02 17:39:57.343964 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.343846    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"openshift-service-ca.crt\""
Nov 02 17:39:57.343964 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.343955    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"default-dockercfg-gsl42\""
Nov 02 17:39:57.344473 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.344456    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"distinct-w1-dc5689-kserve-self-signed-certs\""
Nov 02 17:39:57.353355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.353335    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"]
Nov 02 17:39:57.389769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.389735    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:57.444198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444162    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444218    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv665\" (UniqueName: \"kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444258    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444357    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444384    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444435    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444610    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444719    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.444931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.444898    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.446941 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.446919    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.447298 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.447281    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.504233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.504169    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv665\" (UniqueName: \"kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665\") pod \"distinct-w1-dc5689-kserve-66fdc755cc-wf92n\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.554851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.554776    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:57.956652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:57.956604    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"]
Nov 02 17:39:57.958141 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:39:57.958115    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d97ceb9_8398_4bd7_aea8_174135bcd543.slice/crio-a420d5268a3720749d5ea3d5ef5040fa1ae9076c1814e76e51d3cde3f1a02332 WatchSource:0}: Error finding container a420d5268a3720749d5ea3d5ef5040fa1ae9076c1814e76e51d3cde3f1a02332: Status 404 returned error can't find the container with id a420d5268a3720749d5ea3d5ef5040fa1ae9076c1814e76e51d3cde3f1a02332
Nov 02 17:39:58.807421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:58.807382    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" event={"ID":"8d97ceb9-8398-4bd7-aea8-174135bcd543","Type":"ContainerStarted","Data":"a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65"}
Nov 02 17:39:58.807421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:58.807434    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" event={"ID":"8d97ceb9-8398-4bd7-aea8-174135bcd543","Type":"ContainerStarted","Data":"a420d5268a3720749d5ea3d5ef5040fa1ae9076c1814e76e51d3cde3f1a02332"}
Nov 02 17:39:58.923606 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:58.923551    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" podStartSLOduration=1.9235376770000001 podStartE2EDuration="1.923537677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:58.920057891 +0000 UTC m=+1029.760663825" watchObservedRunningTime="2026-11-02 17:39:58.923537677 +0000 UTC m=+1029.764143642"
Nov 02 17:39:59.066918 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:39:59.066826    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:39:59.555354 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:59.555328    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:39:59.977859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:39:59.977827    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:04.697710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:04.697672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:07.555444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:07.555416    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:40:07.568972 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:07.568942    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:40:07.881726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:07.881697    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 17:40:09.545993 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:09.545961    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:11.352202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:11.352165    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:14.291500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:14.291465    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:19.100706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:19.100665    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:23.862942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:23.862909    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:28.656812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:28.656781    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:29.059479 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:40:29.059393    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:40:30.702235 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:30.702203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:33.365188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:33.365149    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:36.464190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:36.464125    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:36.647391 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:36.647361    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:36.839444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:36.839321    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:38.302147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:38.302093    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:43.388038 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:43.388011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:45.621938 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:45.621900    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:48.178797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:48.178761    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:50.546964 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.546926    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"]
Nov 02 17:40:50.551361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.551332    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.603226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.603190    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a099c3-openshift-default-dockercfg-68w8k\""
Nov 02 17:40:50.641931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.641887    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.641931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.641929    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.641946    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.642079    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.642126    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.642144    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hssd8\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.642247    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.642300    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.642357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.642335    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.659733 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.659687    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"]
Nov 02 17:40:50.743577 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743542    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743587    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743614    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743655    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743673    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743689    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743727    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743772    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.743802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.743795    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hssd8\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.744227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.744140    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.744288 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.744227    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.744333 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.744294    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.744333 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.744321    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.744659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.744619    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.746237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.746214    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.746391 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.746371    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.800106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.800031    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.800308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.800286    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hssd8\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:50.866180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:50.866150    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:51.297704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:51.297677    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"]
Nov 02 17:40:51.298392 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:40:51.298369    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ce98c5_ae32_46e6_a0e8_d40f6621c4dc.slice/crio-465b57c3da1dae5e517b6ae398f99bde0b4ed4bb5a43f14bf6c829cf6093be1b WatchSource:0}: Error finding container 465b57c3da1dae5e517b6ae398f99bde0b4ed4bb5a43f14bf6c829cf6093be1b: Status 404 returned error can't find the container with id 465b57c3da1dae5e517b6ae398f99bde0b4ed4bb5a43f14bf6c829cf6093be1b
Nov 02 17:40:51.300837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:51.300802    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:40:51.300951 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:51.300883    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:40:51.300951 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:51.300927    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:40:52.176881 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.176848    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" event={"ID":"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc","Type":"ContainerStarted","Data":"9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7"}
Nov 02 17:40:52.177288 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.176904    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" event={"ID":"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc","Type":"ContainerStarted","Data":"465b57c3da1dae5e517b6ae398f99bde0b4ed4bb5a43f14bf6c829cf6093be1b"}
Nov 02 17:40:52.345375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.345311    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podStartSLOduration=2.345289402 podStartE2EDuration="2.345289402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:52.341305071 +0000 UTC m=+1083.181911007" watchObservedRunningTime="2026-11-02 17:40:52.345289402 +0000 UTC m=+1083.185895342"
Nov 02 17:40:52.852609 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.852575    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:52.868731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.868700    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:52.869153 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.869130    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:52.869354 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:52.869319    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 17:40:53.867132 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:53.866685    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:53.867132 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:53.866747    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 17:40:54.874082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:54.874049    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:54.874513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:54.874486    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 17:40:55.870797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:55.870750    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:55.871015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:55.870827    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 17:40:56.869696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:56.869072    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:56.869696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:56.869134    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 17:40:57.578817 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:57.578778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:57.866926 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:57.866894    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:40:57.867107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:57.866965    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 17:40:58.873359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:58.873324    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:58.874229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:58.874127    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:58.875347 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:40:58.875322    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:40:59.070038 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:40:59.069999    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:41:02.571415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:02.571364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:07.161600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:07.161563    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:07.358258 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:07.358213    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:12.059369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:12.059330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:16.774217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:16.774171    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:21.453668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:21.453616    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:26.175036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.174975    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:26.443397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.443314    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"]
Nov 02 17:41:26.447187 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.447171    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.497739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.497700    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.497739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.497739    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.497955 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.497839    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.497955 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.497884    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.498042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.498026    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv667\" (UniqueName: \"kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.498080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.498071    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.504158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.504125    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"]
Nov 02 17:41:26.549301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.549271    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"distinct-w3-6e3175-kserve-self-signed-certs\""
Nov 02 17:41:26.593902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.593869    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:26.599602 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.599568    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.599773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.599622    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.599773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.599766    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv667\" (UniqueName: \"kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.599875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.599824    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.599875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.599869    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.599949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.599904    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.600129 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.600105    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.600315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.600288    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.600917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.600895    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.603654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.603606    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.603784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.603681    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.656864 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.656830    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv667\" (UniqueName: \"kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667\") pod \"distinct-w3-6e3175-kserve-599596fc95-dg72l\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:26.760015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:26.759916    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:27.162308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:27.162274    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"]
Nov 02 17:41:27.162680 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:41:27.162432    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8ffa922_9e25_4195_82d3_d7c9f438f65e.slice/crio-de1a9dd4eb71f392304755377cc8136199a7a971ba02537f9ad6ac43b4059661 WatchSource:0}: Error finding container de1a9dd4eb71f392304755377cc8136199a7a971ba02537f9ad6ac43b4059661: Status 404 returned error can't find the container with id de1a9dd4eb71f392304755377cc8136199a7a971ba02537f9ad6ac43b4059661
Nov 02 17:41:27.451664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:27.451540    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" event={"ID":"f8ffa922-9e25-4195-82d3-d7c9f438f65e","Type":"ContainerStarted","Data":"01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864"}
Nov 02 17:41:27.451664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:27.451591    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" event={"ID":"f8ffa922-9e25-4195-82d3-d7c9f438f65e","Type":"ContainerStarted","Data":"de1a9dd4eb71f392304755377cc8136199a7a971ba02537f9ad6ac43b4059661"}
Nov 02 17:41:27.566993 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:27.566940    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" podStartSLOduration=1.566926054 podStartE2EDuration="1.566926054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:27.563792469 +0000 UTC m=+1118.404398403" watchObservedRunningTime="2026-11-02 17:41:27.566926054 +0000 UTC m=+1118.407531989"
Nov 02 17:41:27.760248 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:27.760171    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:29.060973 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:41:29.060938    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:41:30.873778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:30.873683    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:35.576307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:35.576273    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:36.761121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:36.761081    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:36.775142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:36.775109    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:37.527605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:37.527573    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 17:41:40.257425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:40.257391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:44.952942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:44.952896    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:45.969979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:45.969947    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:49.725256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:49.725228    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:51.861251 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:51.861208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:53.993496 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:53.993459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:54.491069 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:54.491040    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:59.060383 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:41:59.060328    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:41:59.242512 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:41:59.242437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:04.079170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:04.079127    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:05.691294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:05.691257    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:08.847156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:08.847107    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:13.571364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:13.571328    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:18.249940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:18.249904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:22.972324 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:22.972287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:27.763294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:27.763249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:29.060296 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:42:29.060254    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:42:30.355393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.355360    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"]
Nov 02 17:42:30.359272 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.359253    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.391790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.391759    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.391935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.391820    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.391980 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.391928    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.392021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.391972    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.392073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.392059    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.392111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.392097    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj9n\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.392144 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.392121    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.392180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.392143    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.392180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.392159    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.465600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.465565    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"]
Nov 02 17:42:30.493797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.493767    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.493974 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.493839    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.493974 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.493871    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.493974 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.493898    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494065    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494150    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj9n\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494206    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494267    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494279    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494324    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494341    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494476    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494509    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.494678 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.494660    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.496790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.496769    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.497329 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.497258    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.550231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.550201    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.550356 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.550306    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj9n\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:30.672945 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:30.672918    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:31.107810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:31.107779    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"]
Nov 02 17:42:31.109747 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:42:31.109726    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277a2900_6b55_4c0a_a759_30f31a5061e2.slice/crio-70ced4102f4a3f96c497d09e3d1bb3d0e3340ff31cea6d928a5caece2fea2ff4 WatchSource:0}: Error finding container 70ced4102f4a3f96c497d09e3d1bb3d0e3340ff31cea6d928a5caece2fea2ff4: Status 404 returned error can't find the container with id 70ced4102f4a3f96c497d09e3d1bb3d0e3340ff31cea6d928a5caece2fea2ff4
Nov 02 17:42:31.112115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:31.112074    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:42:31.112222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:31.112149    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:42:31.112286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:31.112218    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:42:31.911161 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:31.911120    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" event={"ID":"277a2900-6b55-4c0a-a759-30f31a5061e2","Type":"ContainerStarted","Data":"390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b"}
Nov 02 17:42:31.911659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:31.911217    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" event={"ID":"277a2900-6b55-4c0a-a759-30f31a5061e2","Type":"ContainerStarted","Data":"70ced4102f4a3f96c497d09e3d1bb3d0e3340ff31cea6d928a5caece2fea2ff4"}
Nov 02 17:42:32.130675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:32.130608    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podStartSLOduration=2.130593662 podStartE2EDuration="2.130593662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:32.079456683 +0000 UTC m=+1182.920062614" watchObservedRunningTime="2026-11-02 17:42:32.130593662 +0000 UTC m=+1182.971199596"
Nov 02 17:42:32.495128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:32.495077    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:32.673161 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:32.673122    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:32.674370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:32.674347    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:42:32.674496 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:32.674399    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 17:42:33.673885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:33.673852    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:42:33.674387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:33.673904    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 17:42:34.673967 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:34.673921    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:42:34.674542 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:34.674000    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 17:42:35.674349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:35.674315    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:42:35.674780 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:35.674372    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 17:42:36.674137 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:36.674103    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:42:36.674388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:36.674364    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 17:42:37.395354 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:37.395317    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:37.674088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:37.673985    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 17:42:37.674262 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:37.674080    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 17:42:38.677888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:38.677858    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:38.678376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:38.678265    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:38.679166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:38.679141    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:42:42.059672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:42.059608    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:46.310489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:46.310457    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:46.748662 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:46.748597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:49.522509 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:49.522483    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:42:49.526905 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:49.526887    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:42:51.568242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:51.568186    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:55.409588 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:55.409557    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:56.556166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:56.556133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:57.877843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.877811    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"]
Nov 02 17:42:57.881822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.881797    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917218 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917187    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917236    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917336    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917430    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917544    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g78q\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917584    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917727    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.917811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.917787    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:57.933977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.933955    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\""
Nov 02 17:42:57.988696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:57.988662    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"]
Nov 02 17:42:58.019412 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019373    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019433    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019463    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019501    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019545    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019579    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019650    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019687    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g78q\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.019821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019712    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.020029 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.019864    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.020086 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.020065    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.020196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.020159    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.020567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.020546    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.020869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.020844    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.022455 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.022424    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.022700 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.022683    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.177377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.177301    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g78q\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.177377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.177301    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.196266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.196239    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:58.631567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.631543    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"]
Nov 02 17:42:58.633166 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:42:58.633141    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14c7ef8_a016_4ab1_bbae_b1ade638857b.slice/crio-85e6a6da5075074be047d32e6509f3806e0d8459f3a026f886e99bc667f134f3 WatchSource:0}: Error finding container 85e6a6da5075074be047d32e6509f3806e0d8459f3a026f886e99bc667f134f3: Status 404 returned error can't find the container with id 85e6a6da5075074be047d32e6509f3806e0d8459f3a026f886e99bc667f134f3
Nov 02 17:42:58.635588 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.635555    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:42:58.635689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.635646    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:42:58.635689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.635678    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:42:58.685541 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:58.685509    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:59.068243 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:42:59.068164    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:42:59.127835 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:59.127799    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" event={"ID":"b14c7ef8-a016-4ab1-bbae-b1ade638857b","Type":"ContainerStarted","Data":"8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a"}
Nov 02 17:42:59.128012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:59.127867    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" event={"ID":"b14c7ef8-a016-4ab1-bbae-b1ade638857b","Type":"ContainerStarted","Data":"85e6a6da5075074be047d32e6509f3806e0d8459f3a026f886e99bc667f134f3"}
Nov 02 17:42:59.196957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:59.196917    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:42:59.198283 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:59.198264    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:42:59.198345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:59.198325    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:42:59.304953 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:42:59.304895    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podStartSLOduration=2.304879335 podStartE2EDuration="2.304879335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:59.265124663 +0000 UTC m=+1210.105730598" watchObservedRunningTime="2026-11-02 17:42:59.304879335 +0000 UTC m=+1210.145485268"
Nov 02 17:43:00.197143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:00.197111    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:43:00.197618 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:00.197174    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:43:01.196953 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:01.196917    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:43:01.197146 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:01.196975    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:43:01.536657 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:01.536557    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:02.196999 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:02.196967    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:43:02.197200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:02.197026    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:43:02.325177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:02.325147    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:03.197408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:03.197372    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:43:03.197812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:03.197440    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:43:04.196795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:04.196754    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:43:04.197010 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:04.196804    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:43:05.203653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:05.200775    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 17:43:05.203653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:05.200842    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 17:43:06.357716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:06.357669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:07.237799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:07.237750    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:07.238011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:07.237832    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:07.504047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:07.503937    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:43:07.504486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:07.504373    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:43:07.505126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:07.505107    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:43:11.078084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:11.078054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:11.261254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:11.261224    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:11.437554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:11.437515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:12.987747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:12.987715    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"]
Nov 02 17:43:12.991858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:12.991839    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.051622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.051596    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"]
Nov 02 17:43:13.122814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.122781    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.122971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.122827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.122971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.122951    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsxs\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.123077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.122980    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.123077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.123010    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.123177 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.123149    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.123232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.123218    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.123291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.123264    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.123330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.123296    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.223870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.223834    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsxs\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.223870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.223877    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.223913    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.223960    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224010    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224050    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224070    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224136 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224117    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224458 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224154    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224458 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224347    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224526    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224620 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224556    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.224620 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.224597    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.225034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.225013    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.226839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.226818    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.227411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.227387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.281021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.280945    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.281171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.281146    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsxs\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.306259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.306231    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:13.739237 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.739214    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"]
Nov 02 17:43:13.740370 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:43:13.740340    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67d3c42_7784_450a_af12_5e0f6ee2c416.slice/crio-042e364c6c4e04fb77832299768c16cb977fa86fd3ed06590bf854d0be387171 WatchSource:0}: Error finding container 042e364c6c4e04fb77832299768c16cb977fa86fd3ed06590bf854d0be387171: Status 404 returned error can't find the container with id 042e364c6c4e04fb77832299768c16cb977fa86fd3ed06590bf854d0be387171
Nov 02 17:43:13.742756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.742722    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:13.742826 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.742793    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:13.742870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:13.742825    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:14.272885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:14.272837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" event={"ID":"e67d3c42-7784-450a-af12-5e0f6ee2c416","Type":"ContainerStarted","Data":"36988d14b2aa650aaa964cd4f284b7e389b022c86049954e36fc105d6154d06f"}
Nov 02 17:43:14.273275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:14.272951    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" event={"ID":"e67d3c42-7784-450a-af12-5e0f6ee2c416","Type":"ContainerStarted","Data":"042e364c6c4e04fb77832299768c16cb977fa86fd3ed06590bf854d0be387171"}
Nov 02 17:43:14.306442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:14.306414    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:14.308228 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:14.308206    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:14.308332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:14.308249    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:14.480708 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:14.480651    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podStartSLOduration=2.480622071 podStartE2EDuration="2.480622071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:14.478575123 +0000 UTC m=+1225.319181058" watchObservedRunningTime="2026-11-02 17:43:14.480622071 +0000 UTC m=+1225.321228004"
Nov 02 17:43:15.307035 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:15.307005    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:15.307437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:15.307054    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:15.799212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:15.799161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:16.307548 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:16.307512    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:16.307987 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:16.307563    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:17.306849 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:17.306819    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:17.307028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:17.306874    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:18.307367 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:18.307338    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:18.307867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:18.307839    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:19.306740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:19.306700    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:19.307176 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:19.307145    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:19.605445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:19.605333    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:20.310090 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:20.310048    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:20.310407 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:20.310376    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:21.109179 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:21.109144    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:21.309387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:21.309340    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 17:43:21.309731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:21.309690    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 17:43:23.241456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:23.241422    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:23.242004 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:23.241789    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:23.242716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:23.242694    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 17:43:26.077278 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:26.077234    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:29.063099 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:43:29.063053    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:43:30.814438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.814401    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"]
Nov 02 17:43:30.818424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.818405    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.865609 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.865576    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:30.916730 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.916697    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"]
Nov 02 17:43:30.936978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.936942    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937020    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937076    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4w8\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937119    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937155    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937192    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937251    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937283    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:30.937381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:30.937358    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038123 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038089    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038137    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4w8\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038176    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038207    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038548    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038596    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038694    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038731    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.038858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038818    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.039018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038861    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.039018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038863    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.039018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.038966    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.039198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.039175    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.039322 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.039210    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.041184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.041160    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.041698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.041681    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.095300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.095226    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.095454 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.095319    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4w8\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.132312 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.132275    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:31.570195 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.570168    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"]
Nov 02 17:43:31.570975 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:43:31.570951    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15486176_45b2_4801_97c3_b1e54aa96f5e.slice/crio-12bc7228fad19aa159c770aa5fe65c81610b83eb90111f091eb802997d32b690 WatchSource:0}: Error finding container 12bc7228fad19aa159c770aa5fe65c81610b83eb90111f091eb802997d32b690: Status 404 returned error can't find the container with id 12bc7228fad19aa159c770aa5fe65c81610b83eb90111f091eb802997d32b690
Nov 02 17:43:31.573338 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.573309    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:31.573420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.573368    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:31.573420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.573397    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:31.777790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:31.777752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:32.421116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:32.421082    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" event={"ID":"15486176-45b2-4801-97c3-b1e54aa96f5e","Type":"ContainerStarted","Data":"c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6"}
Nov 02 17:43:32.421664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:32.421174    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" event={"ID":"15486176-45b2-4801-97c3-b1e54aa96f5e","Type":"ContainerStarted","Data":"12bc7228fad19aa159c770aa5fe65c81610b83eb90111f091eb802997d32b690"}
Nov 02 17:43:32.589818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:32.589760    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podStartSLOduration=2.589744035 podStartE2EDuration="2.589744035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.587544882 +0000 UTC m=+1243.428150817" watchObservedRunningTime="2026-11-02 17:43:32.589744035 +0000 UTC m=+1243.430349969"
Nov 02 17:43:33.132845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:33.132803    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:33.134441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:33.134408    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:33.134581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:33.134462    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 17:43:34.133254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:34.133223    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:34.133710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:34.133275    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 17:43:35.133529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:35.133491    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:35.134074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:35.133546    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 17:43:35.829120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:35.829076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:36.132696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:36.132662    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:36.132883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:36.132719    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 17:43:37.133600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.133566    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:37.134028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.133650    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 17:43:37.594126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.594045    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"]
Nov 02 17:43:37.598510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.598493    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.704596 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.704553    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6e3175-openshift-default-dockercfg-n7khr\""
Nov 02 17:43:37.710723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.710243    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"]
Nov 02 17:43:37.730396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730362    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730575 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730418    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730575 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730509    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730575    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730614    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730672    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730702    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730757    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.730885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.730819    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpkts\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.831995    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832039    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832056    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832084    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832116    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832151    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832175    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832302 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832206    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832302    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpkts\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832441    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832497    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832604    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.832867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.832715    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.833158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.833132    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.834825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.834796    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.834923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.834906    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.892590 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.892561    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpkts\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.892783 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.892655    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:37.912396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:37.912370    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:38.133140 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.133106    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:38.133314 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.133163    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 17:43:38.348685 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:43:38.348653    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb189342d_b7e0_4f17_bbbc_f5202be41650.slice/crio-819a33964c661854941aab07b8f4e0a6a1d92d090eeeb106367ed8d56485084a WatchSource:0}: Error finding container 819a33964c661854941aab07b8f4e0a6a1d92d090eeeb106367ed8d56485084a: Status 404 returned error can't find the container with id 819a33964c661854941aab07b8f4e0a6a1d92d090eeeb106367ed8d56485084a
Nov 02 17:43:38.349054 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.348705    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"]
Nov 02 17:43:38.351182 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.351151    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:38.351301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.351227    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:38.351301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.351276    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:43:38.492603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.492570    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" event={"ID":"b189342d-b7e0-4f17-bbbc-f5202be41650","Type":"ContainerStarted","Data":"2a234e811549dc5f1d2e19a2765d569921877e7d834f4b8086981ec63ca6f8d8"}
Nov 02 17:43:38.492799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.492704    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" event={"ID":"b189342d-b7e0-4f17-bbbc-f5202be41650","Type":"ContainerStarted","Data":"819a33964c661854941aab07b8f4e0a6a1d92d090eeeb106367ed8d56485084a"}
Nov 02 17:43:38.614509 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.614456    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podStartSLOduration=1.6144396909999998 podStartE2EDuration="1.614439691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:38.611223072 +0000 UTC m=+1249.451829007" watchObservedRunningTime="2026-11-02 17:43:38.614439691 +0000 UTC m=+1249.455045967"
Nov 02 17:43:38.913156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:38.913071    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:39.173396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:39.173285    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:39.174526 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:39.174479    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:39.179972 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:39.179911    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:43:39.915121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:39.915085    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:39.915519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:39.915152    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:40.814227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:40.814188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:40.912790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:40.912757    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:40.912990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:40.912812    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:41.913825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:41.913781    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:41.914275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:41.913846    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:42.913841 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:42.913794    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:42.914253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:42.913880    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:43.923668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:43.920504    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:43.923668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:43.920667    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:44.921565 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:44.921057    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:44.921565 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:44.921156    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:45.326714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:45.325759    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.528654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:45.523800    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.561657 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:45.560753    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" podUID="ef20ad96-c209-4313-8a19-02c3d9b5d999" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.696661 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:45.688812    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" podUID="eb1731d5-2413-48d5-932a-cf777619c149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:45.870653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:45.869711    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" podUID="8d97ceb9-8398-4bd7-aea8-174135bcd543" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 02 17:43:46.100723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.090812    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:46.100723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.090877    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 17:43:46.339654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.335957    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:46.360723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.358802    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:43:46.480655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.476085    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 17:43:46.681864 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.681771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:46.869655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:46.868620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:47.255309 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:47.255273    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:47.255756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:47.255728    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:47.256617 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:47.256596    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:43:51.575360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:51.575320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:56.276397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:43:56.276345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:59.060808 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:43:59.060771    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:44:00.995240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:00.995202    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:06.712417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:06.712380    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:07.486065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:07.485988    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:07.713206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:07.713167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:07.946923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:07.946884    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:10.516915 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:10.516881    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:10.691240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:10.691206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:11.847109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:11.847076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:12.064375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:12.064341    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:12.760369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:12.760332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:13.473805 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.473773    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"]
Nov 02 17:44:13.478217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.478197    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503289    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503338    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503504    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503623 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503539    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503662    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkbw\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503861    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.503911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.503889    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.583186 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.583144    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"]
Nov 02 17:44:13.604603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604559    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkbw\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.604603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604593    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.604860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604618    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.604860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604696    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.604860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604776    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.604860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604815    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.604860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604837    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605129 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604869    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605129 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.604922    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.605166    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.605473    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.605482    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.605552    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.605885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.605834    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.607664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.607639    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.607900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.607877    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.660982 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.660949    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.661130 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.661000    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkbw\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.662167 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.662140    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:13.792782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.792707    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:13.872932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:13.872893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:14.250839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:14.250809    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"]
Nov 02 17:44:14.251895 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:44:14.251871    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894105b3_ee9b_48e1_979e_f57fb8748065.slice/crio-d3151dd1e775b77308263a38fdbf221fde40ae23abefaa3cbe1097a9f7dc6469 WatchSource:0}: Error finding container d3151dd1e775b77308263a38fdbf221fde40ae23abefaa3cbe1097a9f7dc6469: Status 404 returned error can't find the container with id d3151dd1e775b77308263a38fdbf221fde40ae23abefaa3cbe1097a9f7dc6469
Nov 02 17:44:14.254159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:14.254118    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:44:14.254264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:14.254200    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:44:14.254264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:14.254233    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:44:15.270328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:15.270285    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" event={"ID":"894105b3-ee9b-48e1-979e-f57fb8748065","Type":"ContainerStarted","Data":"c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6"}
Nov 02 17:44:15.270769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:15.270376    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" event={"ID":"894105b3-ee9b-48e1-979e-f57fb8748065","Type":"ContainerStarted","Data":"d3151dd1e775b77308263a38fdbf221fde40ae23abefaa3cbe1097a9f7dc6469"}
Nov 02 17:44:15.472815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:15.472764    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podStartSLOduration=2.472747707 podStartE2EDuration="2.472747707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:15.435315053 +0000 UTC m=+1286.275920986" watchObservedRunningTime="2026-11-02 17:44:15.472747707 +0000 UTC m=+1286.313353715"
Nov 02 17:44:15.792707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:15.792673    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:15.794558 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:15.794528    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:15.794711 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:15.794591    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 17:44:16.793538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:16.793505    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:16.793981 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:16.793565    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 17:44:16.925200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:16.925154    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:17.161747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:17.161707    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:17.793466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:17.793434    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:17.793720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:17.793697    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 17:44:18.494109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:18.494068    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:18.793530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:18.793435    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:18.793530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:18.793487    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 17:44:19.793311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:19.793276    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:19.793530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:19.793338    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 17:44:20.793284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:20.793236    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:20.793778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:20.793309    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 17:44:21.797969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:21.797931    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:21.798474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:21.798450    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:21.799350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:21.799329    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:44:22.912729 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:22.912692    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:22.988896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:22.988865    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"]
Nov 02 17:44:22.993020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:22.993000    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054141 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054107    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054312 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054169    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054301    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054418    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054498    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054646    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054709    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054762    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.054855 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.054807    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svq4h\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.097504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.097466    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"]
Nov 02 17:44:23.122800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.122772    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:23.156160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156115    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156200    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156268    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156355    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156387    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156558    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156658    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svq4h\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.156723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.156672    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.157432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.157399    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.157691 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.157663    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.158104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.158079    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.163769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.158459    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.163769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.158551    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.163769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.159201    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.163769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.159370    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.174268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.174239    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.212490 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.212463    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.214809 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.214785    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svq4h\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.307576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.307545    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:23.498108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.498019    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:23.743358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.743315    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"]
Nov 02 17:44:23.751283 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:44:23.751220    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821b43c6_579c_4af9_9328_1d3d113830ee.slice/crio-c8259e90aef0ba984d9c51b963ca358f4540cb0af50805d7b8a672a7ffe3d1f0 WatchSource:0}: Error finding container c8259e90aef0ba984d9c51b963ca358f4540cb0af50805d7b8a672a7ffe3d1f0: Status 404 returned error can't find the container with id c8259e90aef0ba984d9c51b963ca358f4540cb0af50805d7b8a672a7ffe3d1f0
Nov 02 17:44:23.753903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.753872    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:44:23.753982 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.753940    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:44:23.753982 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:23.753969    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:44:24.354611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:24.354560    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" event={"ID":"821b43c6-579c-4af9-9328-1d3d113830ee","Type":"ContainerStarted","Data":"532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113"}
Nov 02 17:44:24.355143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:24.354666    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" event={"ID":"821b43c6-579c-4af9-9328-1d3d113830ee","Type":"ContainerStarted","Data":"c8259e90aef0ba984d9c51b963ca358f4540cb0af50805d7b8a672a7ffe3d1f0"}
Nov 02 17:44:24.519755 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:24.519697    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podStartSLOduration=2.519677147 podStartE2EDuration="2.519677147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:24.51809462 +0000 UTC m=+1295.358700556" watchObservedRunningTime="2026-11-02 17:44:24.519677147 +0000 UTC m=+1295.360283081"
Nov 02 17:44:25.308489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:25.308448    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:25.310184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:25.310159    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:44:25.310316 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:25.310203    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 17:44:26.308713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:26.308678    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:44:26.309121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:26.308744    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 17:44:27.308425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:27.308386    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:44:27.308589 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:27.308452    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 17:44:28.308059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:28.308020    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:44:28.308444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:28.308084    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 17:44:28.393292 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:28.393242    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:29.087141 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:44:29.087099    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:44:29.308652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:29.308596    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:44:29.309185 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:29.308668    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 17:44:29.588027 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:29.587993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:30.308219 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:30.308177    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 17:44:30.308401 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:30.308256    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 17:44:32.311311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:32.310715    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:32.311311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:32.310775    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:33.357654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:33.356730    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:33.357654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:33.356811    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:33.791232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:33.791163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:33.982779 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:33.982748    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:34.117905 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:34.117862    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:34.119375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:34.119353    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:34.121383 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:34.121366    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:44:34.169533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:34.169499    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:35.811159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:35.811126    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:38.428492 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:38.428450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:38.646508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:38.646465    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:39.383984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.383935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:39.435464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.435430    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"]
Nov 02 17:44:39.439317 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.439296    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.541285 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.541249    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2296d33cdbd2\""
Nov 02 17:44:39.541522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.541330    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 17:44:39.541670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.541436    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 17:44:39.541939 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.541830    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cvktf\""
Nov 02 17:44:39.541939 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.541897    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 17:44:39.541939 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.541915    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 17:44:39.548498 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.548463    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"]
Nov 02 17:44:39.574229 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.574191    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68k8z\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.574417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.574334    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.574641 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.574596    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.574747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.574694    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.676357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.676256    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.676357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.676312    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.676357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.676335    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68k8z\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.676673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.676382    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.677209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.677182    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.677310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.677221    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.678991 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.678966    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.733887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.733843    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68k8z\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z\") pod \"payload-processing-external-model-5d5b9f745f-v78d7\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") " pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:39.758688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:39.758621    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:40.147717 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:40.147678    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"]
Nov 02 17:44:40.727466 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:40.727429    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" event={"ID":"1941071b-deda-498f-97b4-02f2cbede2f4","Type":"ContainerStarted","Data":"5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480"}
Nov 02 17:44:40.728011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:40.727524    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" event={"ID":"1941071b-deda-498f-97b4-02f2cbede2f4","Type":"ContainerStarted","Data":"4da630f38380f4ff2250b9e1afa4f26a3f5097b4a3b84a30d411237d49b54b75"}
Nov 02 17:44:40.728011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:40.727664    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:40.843819 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:40.843749    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" podStartSLOduration=1.843728901 podStartE2EDuration="1.843728901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:40.840595433 +0000 UTC m=+1311.681201367" watchObservedRunningTime="2026-11-02 17:44:40.843728901 +0000 UTC m=+1311.684334837"
Nov 02 17:44:43.930501 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:43.930469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:44.121154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:44.121117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:44.314250 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:44.314160    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:47.192355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:47.192322    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:47.377041 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:47.377012    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:48.958578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:48.958527    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:51.201394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:51.201362    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:51.745338 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:51.745285    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:44:51.871360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:51.871329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:52.987560 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:52.987520    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"]
Nov 02 17:44:52.988920 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:52.988889    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:44:54.266442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:54.266406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:55.259148 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:55.259078    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:56.880675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:56.880613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:57.069044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:57.069010    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:57.253922 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:57.253811    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:57.988593 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:57.988542    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" containerName="payload-processing" containerID="cri-o://5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480" gracePeriod=25
Nov 02 17:44:58.265939 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.265905    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:58.327705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.327671    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials\") pod \"1941071b-deda-498f-97b4-02f2cbede2f4\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") "
Nov 02 17:44:58.327869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.327753    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume\") pod \"1941071b-deda-498f-97b4-02f2cbede2f4\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") "
Nov 02 17:44:58.327963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.327889    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume\") pod \"1941071b-deda-498f-97b4-02f2cbede2f4\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") "
Nov 02 17:44:58.327963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.327929    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z\") pod \"1941071b-deda-498f-97b4-02f2cbede2f4\" (UID: \"1941071b-deda-498f-97b4-02f2cbede2f4\") "
Nov 02 17:44:58.328258 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.328215    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume" pod "1941071b-deda-498f-97b4-02f2cbede2f4" (UID: "1941071b-deda-498f-97b4-02f2cbede2f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:58.328258 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.328221    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume" pod "1941071b-deda-498f-97b4-02f2cbede2f4" (UID: "1941071b-deda-498f-97b4-02f2cbede2f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:58.328444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.328339    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:58.328444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.328357    4476 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1941071b-deda-498f-97b4-02f2cbede2f4-routing-overlay-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:58.330164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.330138    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials" pod "1941071b-deda-498f-97b4-02f2cbede2f4" (UID: "1941071b-deda-498f-97b4-02f2cbede2f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:58.330253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.330170    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z" pod "1941071b-deda-498f-97b4-02f2cbede2f4" (UID: "1941071b-deda-498f-97b4-02f2cbede2f4"). InnerVolumeSpecName "kube-api-access-68k8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:58.430104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.430067    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68k8z\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-kube-api-access-68k8z\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:58.430104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.430104    4476 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1941071b-deda-498f-97b4-02f2cbede2f4-provider-credentials\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:44:58.884476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.884436    4476 generic.go:356] "Generic (PLEG): container finished" podID="1941071b-deda-498f-97b4-02f2cbede2f4" containerID="5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480" exitCode=0
Nov 02 17:44:58.884683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.884485    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"
Nov 02 17:44:58.884683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.884520    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" event={"ID":"1941071b-deda-498f-97b4-02f2cbede2f4","Type":"ContainerDied","Data":"5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480"}
Nov 02 17:44:58.884683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.884578    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7" event={"ID":"1941071b-deda-498f-97b4-02f2cbede2f4","Type":"ContainerDied","Data":"4da630f38380f4ff2250b9e1afa4f26a3f5097b4a3b84a30d411237d49b54b75"}
Nov 02 17:44:58.884683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.884606    4476 scope.go:122] "RemoveContainer" containerID="5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480"
Nov 02 17:44:58.895971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.895954    4476 scope.go:122] "RemoveContainer" containerID="5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480"
Nov 02 17:44:58.896228 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:44:58.896205    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480\": container with ID starting with 5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480 not found: ID does not exist" containerID="5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480"
Nov 02 17:44:58.896274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:58.896237    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480"} err="failed to get container status \"5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480\": rpc error: code = NotFound desc = could not find container \"5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480\": container with ID starting with 5cf5580f85d2baeec1bf7a1766c4a481d155a19eca037185da2659a9d2800480 not found: ID does not exist"
Nov 02 17:44:59.048559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:59.048513    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"]
Nov 02 17:44:59.062032 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:44:59.061986    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:44:59.097947 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:59.097910    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5b9f745f-v78d7"]
Nov 02 17:44:59.386252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:59.386219    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" path="/var/lib/kubelet/pods/1941071b-deda-498f-97b4-02f2cbede2f4/volumes"
Nov 02 17:44:59.511747 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:44:59.511715    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:00.135013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.134953    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:00.172737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.172699    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"]
Nov 02 17:45:00.173268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.173250    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" containerName="payload-processing"
Nov 02 17:45:00.173268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.173265    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" containerName="payload-processing"
Nov 02 17:45:00.173443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.173348    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1941071b-deda-498f-97b4-02f2cbede2f4" containerName="payload-processing"
Nov 02 17:45:00.178109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.178087    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.189164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.189144    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"]
Nov 02 17:45:00.192745 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.192724    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.238651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.229927    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 17:45:00.257655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.254408    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.257655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.254470    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.257655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.254530    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm28g\" (UniqueName: \"kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.257655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.254685    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97z44\" (UniqueName: \"kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.312658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.311424    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"]
Nov 02 17:45:00.360415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.359358    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.360415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.359431    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28g\" (UniqueName: \"kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.360415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.359569    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97z44\" (UniqueName: \"kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.360415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.359732    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.367375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.360524    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.367375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.361239    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.367375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.363263    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"]
Nov 02 17:45:00.367375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.364165    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:00.439392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.439289    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm28g\" (UniqueName: \"kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.462198 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.458447    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97z44\" (UniqueName: \"kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:00.584015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.583978    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:00.699721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.696923    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"]
Nov 02 17:45:00.717715 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.717601    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:00.827702 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.827674    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f8d61e-dockercfg-s4b8t\""
Nov 02 17:45:00.890171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.890142    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f8d61e\""
Nov 02 17:45:00.956090 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.956006    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"]
Nov 02 17:45:00.958750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.958709    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"]
Nov 02 17:45:00.986678 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:00.986358    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:01.124655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.089122    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"]
Nov 02 17:45:01.124655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.089249    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.199650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.197903    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grz7\" (UniqueName: \"kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7\") pod \"payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.199650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.198022    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.199650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.198112    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.199650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.198150    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpw68\" (UniqueName: \"kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68\") pod \"payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.303789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.299400    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grz7\" (UniqueName: \"kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7\") pod \"payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.303789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.299556    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.303789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.299679    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.303789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.299723    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpw68\" (UniqueName: \"kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68\") pod \"payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.303789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.301512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.303789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.302268    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.405660 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.400247    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpw68\" (UniqueName: \"kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68\") pod \"payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.417524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.417454    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grz7\" (UniqueName: \"kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7\") pod \"payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.442960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.442889    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:01.549670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.537776    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"]
Nov 02 17:45:01.549670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.538064    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:01.549670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.543003    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:01.549670 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:45:01.548006    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12323335_16c2_4f16_b6f4_1cb1b0b71c08.slice/crio-cb184e22e9fe5aa8e7e61a271231c7d3181d6da950fd8823b4890b94e31c67ef WatchSource:0}: Error finding container cb184e22e9fe5aa8e7e61a271231c7d3181d6da950fd8823b4890b94e31c67ef: Status 404 returned error can't find the container with id cb184e22e9fe5aa8e7e61a271231c7d3181d6da950fd8823b4890b94e31c67ef
Nov 02 17:45:01.816207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:01.806311    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"]
Nov 02 17:45:02.031728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:02.031702    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"]
Nov 02 17:45:02.153666 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:02.152809    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" event={"ID":"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2","Type":"ContainerStarted","Data":"a6ccbcd2949052904ce6cbec829254b2ffda18bae85120b41a4b342999229a1a"}
Nov 02 17:45:02.232740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:02.231844    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" event={"ID":"12323335-16c2-4f16-b6f4-1cb1b0b71c08","Type":"ContainerStarted","Data":"cb184e22e9fe5aa8e7e61a271231c7d3181d6da950fd8823b4890b94e31c67ef"}
Nov 02 17:45:02.690705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:02.678704    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"]
Nov 02 17:45:03.333641 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.333584    4476 generic.go:356] "Generic (PLEG): container finished" podID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerID="7ee9398cba240c15f23ebda9b4762eea9ac0a7412a4470f85d17dd12d481cf0a" exitCode=60
Nov 02 17:45:03.334112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.333738    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" event={"ID":"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2","Type":"ContainerDied","Data":"7ee9398cba240c15f23ebda9b4762eea9ac0a7412a4470f85d17dd12d481cf0a"}
Nov 02 17:45:03.335406 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.335224    4476 scope.go:122] "RemoveContainer" containerID="7ee9398cba240c15f23ebda9b4762eea9ac0a7412a4470f85d17dd12d481cf0a"
Nov 02 17:45:03.335786 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.335747    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r" event={"ID":"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67","Type":"ContainerStarted","Data":"6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289"}
Nov 02 17:45:03.335867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.335849    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r" event={"ID":"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67","Type":"ContainerStarted","Data":"ddb7d62f154bb8ce778ac4221ff85a2af2e87ab79ad570d04a44973fc3fe622e"}
Nov 02 17:45:03.336737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.336457    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:03.348231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.348201    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb" event={"ID":"b325e916-7c24-4814-ab7b-1287b7b8e524","Type":"ContainerStarted","Data":"594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d"}
Nov 02 17:45:03.349103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.349076    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb" event={"ID":"b325e916-7c24-4814-ab7b-1287b7b8e524","Type":"ContainerStarted","Data":"1814e1bffa1b66ed0cd6dd7b6aec06e7efcd6bbf474c390b98e9bea9be22a443"}
Nov 02 17:45:03.349851 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.349831    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:03.353766 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.353742    4476 generic.go:356] "Generic (PLEG): container finished" podID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerID="29f2d7d6a94d384aaec050c86aa564e0bf1ca5b0d72406553e9e206e26b82330" exitCode=60
Nov 02 17:45:03.353927 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.353908    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" event={"ID":"12323335-16c2-4f16-b6f4-1cb1b0b71c08","Type":"ContainerDied","Data":"29f2d7d6a94d384aaec050c86aa564e0bf1ca5b0d72406553e9e206e26b82330"}
Nov 02 17:45:03.354376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.354358    4476 scope.go:122] "RemoveContainer" containerID="29f2d7d6a94d384aaec050c86aa564e0bf1ca5b0d72406553e9e206e26b82330"
Nov 02 17:45:03.727475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.727420    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb" podStartSLOduration=3.727402683 podStartE2EDuration="3.727402683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.723438953 +0000 UTC m=+1334.564044887" watchObservedRunningTime="2026-11-02 17:45:03.727402683 +0000 UTC m=+1334.568008618"
Nov 02 17:45:03.839152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:03.839100    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r" podStartSLOduration=3.839085969 podStartE2EDuration="3.839085969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.83643691 +0000 UTC m=+1334.677042849" watchObservedRunningTime="2026-11-02 17:45:03.839085969 +0000 UTC m=+1334.679691902"
Nov 02 17:45:04.368246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:04.368206    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" event={"ID":"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2","Type":"ContainerStarted","Data":"d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3"}
Nov 02 17:45:04.370544 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:04.370518    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" event={"ID":"12323335-16c2-4f16-b6f4-1cb1b0b71c08","Type":"ContainerStarted","Data":"29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977"}
Nov 02 17:45:04.491643 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:04.491565    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" podStartSLOduration=4.49154521 podStartE2EDuration="4.49154521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:04.487171092 +0000 UTC m=+1335.327777027" watchObservedRunningTime="2026-11-02 17:45:04.49154521 +0000 UTC m=+1335.332151145"
Nov 02 17:45:04.598837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:04.598772    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" podStartSLOduration=4.598751004 podStartE2EDuration="4.598751004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:04.598194026 +0000 UTC m=+1335.438799959" watchObservedRunningTime="2026-11-02 17:45:04.598751004 +0000 UTC m=+1335.439356940"
Nov 02 17:45:05.379131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.379096    4476 generic.go:356] "Generic (PLEG): container finished" podID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerID="d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3" exitCode=60
Nov 02 17:45:05.379681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.379658    4476 scope.go:122] "RemoveContainer" containerID="d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3"
Nov 02 17:45:05.380037 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:05.380014    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2_odh-ai-gateway-infra(8f8f91a4-0e71-4a3f-91d9-9f0528d959e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2"
Nov 02 17:45:05.382031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.382001    4476 generic.go:356] "Generic (PLEG): container finished" podID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerID="29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977" exitCode=60
Nov 02 17:45:05.382571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.382447    4476 scope.go:122] "RemoveContainer" containerID="29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977"
Nov 02 17:45:05.383288 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:05.383251    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg_odh-ai-gateway-infra(12323335-16c2-4f16-b6f4-1cb1b0b71c08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08"
Nov 02 17:45:05.388259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.388232    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" event={"ID":"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2","Type":"ContainerDied","Data":"d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3"}
Nov 02 17:45:05.388370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.388297    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" event={"ID":"12323335-16c2-4f16-b6f4-1cb1b0b71c08","Type":"ContainerDied","Data":"29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977"}
Nov 02 17:45:05.388370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.388338    4476 scope.go:122] "RemoveContainer" containerID="7ee9398cba240c15f23ebda9b4762eea9ac0a7412a4470f85d17dd12d481cf0a"
Nov 02 17:45:05.417350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.417322    4476 scope.go:122] "RemoveContainer" containerID="29f2d7d6a94d384aaec050c86aa564e0bf1ca5b0d72406553e9e206e26b82330"
Nov 02 17:45:05.831408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:05.831332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:06.400378 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:06.400286    4476 scope.go:122] "RemoveContainer" containerID="d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3"
Nov 02 17:45:06.400919 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:06.400643    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2_odh-ai-gateway-infra(8f8f91a4-0e71-4a3f-91d9-9f0528d959e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2"
Nov 02 17:45:06.402516 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:06.402493    4476 scope.go:122] "RemoveContainer" containerID="29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977"
Nov 02 17:45:06.402837 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:06.402810    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg_odh-ai-gateway-infra(12323335-16c2-4f16-b6f4-1cb1b0b71c08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08"
Nov 02 17:45:10.371913 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:10.371876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:14.374047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:14.374013    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:45:15.195083 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:15.195048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:15.383415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:15.383386    4476 scope.go:122] "RemoveContainer" containerID="29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977"
Nov 02 17:45:15.383830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:15.383607    4476 scope.go:122] "RemoveContainer" containerID="d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3"
Nov 02 17:45:15.386518 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:15.386502    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:45:16.489263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.489176    4476 generic.go:356] "Generic (PLEG): container finished" podID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerID="66ea8a96e72cd6aa2ab8c2db6a803c9c4db4da0c547f584c93fd8cf53da59996" exitCode=60
Nov 02 17:45:16.489736 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.489250    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" event={"ID":"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2","Type":"ContainerDied","Data":"66ea8a96e72cd6aa2ab8c2db6a803c9c4db4da0c547f584c93fd8cf53da59996"}
Nov 02 17:45:16.489736 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.489344    4476 scope.go:122] "RemoveContainer" containerID="d538fc26321e86ed226dcc038de067e0cacef27a23a6870ff24fef484d4b5de3"
Nov 02 17:45:16.489883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.489779    4476 scope.go:122] "RemoveContainer" containerID="66ea8a96e72cd6aa2ab8c2db6a803c9c4db4da0c547f584c93fd8cf53da59996"
Nov 02 17:45:16.490087 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:16.490064    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2_odh-ai-gateway-infra(8f8f91a4-0e71-4a3f-91d9-9f0528d959e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2"
Nov 02 17:45:16.491806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.491778    4476 generic.go:356] "Generic (PLEG): container finished" podID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerID="a30a17a33c48340e9c7035a50ec773976bc102bf48c46ff8875686ff5fd502f9" exitCode=60
Nov 02 17:45:16.491911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.491850    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" event={"ID":"12323335-16c2-4f16-b6f4-1cb1b0b71c08","Type":"ContainerDied","Data":"a30a17a33c48340e9c7035a50ec773976bc102bf48c46ff8875686ff5fd502f9"}
Nov 02 17:45:16.492197 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.492181    4476 scope.go:122] "RemoveContainer" containerID="a30a17a33c48340e9c7035a50ec773976bc102bf48c46ff8875686ff5fd502f9"
Nov 02 17:45:16.492498 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:16.492474    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg_odh-ai-gateway-infra(12323335-16c2-4f16-b6f4-1cb1b0b71c08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08"
Nov 02 17:45:16.503472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:16.503452    4476 scope.go:122] "RemoveContainer" containerID="29b9f79f9f2b524367121f01125a521b309556e07c8d468533a29b7649ea5977"
Nov 02 17:45:17.616224 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.616190    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"]
Nov 02 17:45:17.769088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.769057    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"]
Nov 02 17:45:17.784252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.784227    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:17.920038 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.920010    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca\") pod \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") "
Nov 02 17:45:17.920205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.920133    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g\" (UniqueName: \"kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g\") pod \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\" (UID: \"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2\") "
Nov 02 17:45:17.920400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.920379    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca" pod "8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" (UID: "8f8f91a4-0e71-4a3f-91d9-9f0528d959e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:17.920591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.920578    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:45:17.922478 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.922455    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g" pod "8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" (UID: "8f8f91a4-0e71-4a3f-91d9-9f0528d959e2"). InnerVolumeSpecName "kube-api-access-lm28g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:17.952002 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:17.951977    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:18.021251 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.021218    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca\") pod \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") "
Nov 02 17:45:18.021443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.021432    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44\" (UniqueName: \"kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44\") pod \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\" (UID: \"12323335-16c2-4f16-b6f4-1cb1b0b71c08\") "
Nov 02 17:45:18.021665 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.021606    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca" pod "12323335-16c2-4f16-b6f4-1cb1b0b71c08" (UID: "12323335-16c2-4f16-b6f4-1cb1b0b71c08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:18.021820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.021800    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm28g\" (UniqueName: \"kubernetes.io/projected/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2-kube-api-access-lm28g\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.021882 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.021827    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12323335-16c2-4f16-b6f4-1cb1b0b71c08-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.023793 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.023766    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44" pod "12323335-16c2-4f16-b6f4-1cb1b0b71c08" (UID: "12323335-16c2-4f16-b6f4-1cb1b0b71c08"). InnerVolumeSpecName "kube-api-access-97z44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:18.123067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.123037    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97z44\" (UniqueName: \"kubernetes.io/projected/12323335-16c2-4f16-b6f4-1cb1b0b71c08-kube-api-access-97z44\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.512663 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.512618    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"
Nov 02 17:45:18.512854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.512675    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2" event={"ID":"8f8f91a4-0e71-4a3f-91d9-9f0528d959e2","Type":"ContainerDied","Data":"a6ccbcd2949052904ce6cbec829254b2ffda18bae85120b41a4b342999229a1a"}
Nov 02 17:45:18.512854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.512745    4476 scope.go:122] "RemoveContainer" containerID="66ea8a96e72cd6aa2ab8c2db6a803c9c4db4da0c547f584c93fd8cf53da59996"
Nov 02 17:45:18.514500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.514462    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"
Nov 02 17:45:18.514500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.514469    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg" event={"ID":"12323335-16c2-4f16-b6f4-1cb1b0b71c08","Type":"ContainerDied","Data":"cb184e22e9fe5aa8e7e61a271231c7d3181d6da950fd8823b4890b94e31c67ef"}
Nov 02 17:45:18.525337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.525318    4476 scope.go:122] "RemoveContainer" containerID="a30a17a33c48340e9c7035a50ec773976bc102bf48c46ff8875686ff5fd502f9"
Nov 02 17:45:18.680600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.680566    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"]
Nov 02 17:45:18.687212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.687176    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894025-bvqf2"]
Nov 02 17:45:18.840375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.840290    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"]
Nov 02 17:45:18.843888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:18.843862    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894025-sv8bg"]
Nov 02 17:45:19.268640 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.268606    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"]
Nov 02 17:45:19.269168 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269154    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:19.269213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269170    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:19.269213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269189    4476 cpu_manager.go:414] "containerMap: removing container" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:19.269213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269193    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:19.269213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269211    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:19.269337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269217    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:19.269337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269234    4476 cpu_manager.go:414] "containerMap: removing container" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:19.269337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269239    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:19.269337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269321    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:19.269337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269330    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:19.269337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269336    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:19.269520 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.269342    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:19.273726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.273711    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.326958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.326924    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"openshift-service-ca.crt\""
Nov 02 17:45:19.329797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.329771    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"]
Nov 02 17:45:19.341429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.341390    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"kube-root-ca.crt\""
Nov 02 17:45:19.373711 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.373683    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"default-dockercfg-kdg69\""
Nov 02 17:45:19.373878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.373778    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"distinct-w0-ceb1ae-kserve-self-signed-certs\""
Nov 02 17:45:19.383392 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.383367    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" path="/var/lib/kubelet/pods/12323335-16c2-4f16-b6f4-1cb1b0b71c08/volumes"
Nov 02 17:45:19.383800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.383787    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" path="/var/lib/kubelet/pods/8f8f91a4-0e71-4a3f-91d9-9f0528d959e2/volumes"
Nov 02 17:45:19.395587 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.395553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:19.439796 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.439765    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.440012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.439832    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.440012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.439873    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.440124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.440024    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.440165 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.440151    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.440231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.440206    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxng\" (UniqueName: \"kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.541561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541466    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.541750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541572    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.541750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541602    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxng\" (UniqueName: \"kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.541750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541698    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.541917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541756    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.541917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541777    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.542016 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.541923    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.542178 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.542149    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.542689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.542660    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.545883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.544719    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.545883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.545365    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.598902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.598870    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxng\" (UniqueName: \"kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng\") pod \"distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:19.886443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:19.886416    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:20.003971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:20.003937    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:20.271538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:20.271513    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"]
Nov 02 17:45:20.272880 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:45:20.272850    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44f022ae_9bf4_4ecd_a105_08b66edf63ee.slice/crio-518044bec8e90450aea4c96f3e21891c6d63dcd209b3e1a09b409efee1c522f7 WatchSource:0}: Error finding container 518044bec8e90450aea4c96f3e21891c6d63dcd209b3e1a09b409efee1c522f7: Status 404 returned error can't find the container with id 518044bec8e90450aea4c96f3e21891c6d63dcd209b3e1a09b409efee1c522f7
Nov 02 17:45:20.535087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:20.534992    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8" event={"ID":"44f022ae-9bf4-4ecd-a105-08b66edf63ee","Type":"ContainerStarted","Data":"676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56"}
Nov 02 17:45:20.535087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:20.535043    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8" event={"ID":"44f022ae-9bf4-4ecd-a105-08b66edf63ee","Type":"ContainerStarted","Data":"518044bec8e90450aea4c96f3e21891c6d63dcd209b3e1a09b409efee1c522f7"}
Nov 02 17:45:20.649162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:20.649102    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8" podStartSLOduration=1.649087363 podStartE2EDuration="1.649087363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:20.64693174 +0000 UTC m=+1351.487537674" watchObservedRunningTime="2026-11-02 17:45:20.649087363 +0000 UTC m=+1351.489693328"
Nov 02 17:45:20.886791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:20.886758    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:24.964097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:24.964058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:29.062538 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:29.062485    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:45:29.882933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:29.882901    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:29.886756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:29.886729    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:29.901357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:29.901333    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:30.067565 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:30.067457    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:30.612969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:30.612940    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 17:45:34.593206 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.593171    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"]
Nov 02 17:45:34.594553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.594526    4476 cpu_manager.go:414] "containerMap: removing container" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:34.594750 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.594736    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:34.594936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.594924    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:34.595028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.595017    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:34.595264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.595252    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="12323335-16c2-4f16-b6f4-1cb1b0b71c08" containerName="cleanup"
Nov 02 17:45:34.595370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.595360    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8f91a4-0e71-4a3f-91d9-9f0528d959e2" containerName="cleanup"
Nov 02 17:45:34.609898 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.609873    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.653166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.653129    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"]
Nov 02 17:45:34.653397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.653368    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.653490 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.653469    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.653562 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.653531    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cht\" (UniqueName: \"kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.653655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.653610    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.693651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.692210    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f8d61e-serving-cert\""
Nov 02 17:45:34.780688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.780245    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.780688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.780331    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cht\" (UniqueName: \"kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.780688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.780408    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.780688 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.780504    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.789652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.787720    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.789652 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:34.787829    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-serving-cert: secret "maas-api-e2e-api-a-f8d61e-serving-cert" not found
Nov 02 17:45:34.789652 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:34.787886    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls podName:c6eeef77-9a60-47e0-8d06-977ca9764adf nodeName:}" failed. No retries permitted until 2026-11-02 17:45:35.287866595 +0000 UTC m=+1366.128472512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls") pod "maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" (UID: "c6eeef77-9a60-47e0-8d06-977ca9764adf") : secret "maas-api-e2e-api-a-f8d61e-serving-cert" not found
Nov 02 17:45:34.820391 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.803489    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:34.830963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.830930    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:34.853411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:34.853336    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cht\" (UniqueName: \"kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:35.050654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:35.048081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:35.253388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:35.253346    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:35.299325 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:35.299289    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:35.317728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:35.316226    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\") pod \"maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:35.565706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:35.564829    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:36.263656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:36.249936    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"]
Nov 02 17:45:36.269434 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:45:36.269391    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eeef77_9a60_47e0_8d06_977ca9764adf.slice/crio-2fed0c69f3dec6ec9ea17a53c1913bf70c206d800bd010da392a269a86e0bf05 WatchSource:0}: Error finding container 2fed0c69f3dec6ec9ea17a53c1913bf70c206d800bd010da392a269a86e0bf05: Status 404 returned error can't find the container with id 2fed0c69f3dec6ec9ea17a53c1913bf70c206d800bd010da392a269a86e0bf05
Nov 02 17:45:36.275409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:36.273094    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:45:36.790066 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:36.789961    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" event={"ID":"c6eeef77-9a60-47e0-8d06-977ca9764adf","Type":"ContainerStarted","Data":"2fed0c69f3dec6ec9ea17a53c1913bf70c206d800bd010da392a269a86e0bf05"}
Nov 02 17:45:37.799172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:37.799133    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" event={"ID":"c6eeef77-9a60-47e0-8d06-977ca9764adf","Type":"ContainerStarted","Data":"52e6e4a4aaad39b6ddd2f80ac99372be05a13299fefd35a2f6f7c975e8cc5eb5"}
Nov 02 17:45:37.799603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:37.799222    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:37.959019 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:37.958963    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" podStartSLOduration=2.990261583 podStartE2EDuration="3.958946883s" totalImagesPullingTime="968.6853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:37.956490982 +0000 UTC m=+1368.797096915" watchObservedRunningTime="2026-11-02 17:45:37.958946883 +0000 UTC m=+1368.799552817"
Nov 02 17:45:40.186050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:40.186011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:43.812520 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:43.812494    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:45:44.972770 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:44.972732    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:47.521220 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:47.521187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:49.678977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:49.678941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:54.378808 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:54.378775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:59.066506 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:45:59.066456    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:45:59.072753 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:45:59.072591    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:02.734791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:02.734696    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:03.879076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:03.879024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:05.922367 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:05.922335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:06.140651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:06.130780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:08.804804 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:08.804713    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:11.261869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:11.261832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:13.690458 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:13.690428    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:17.092497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:17.092466    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:18.580381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:18.579343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:23.506824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:23.506782    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:23.711307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:23.711279    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:24.403082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:24.402989    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:25.013307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:25.013269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:25.224225 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:25.224179    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:26.073430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:26.073400    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:26.720938 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:26.720761    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:27.011397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:27.011287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:28.945001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:28.944962    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:29.061014 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:46:29.060974    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:46:30.223242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:30.223206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:30.429107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:30.429073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:31.090503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:31.090470    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:31.347673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:31.347566    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:31.607958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:31.607864    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:33.530342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:33.530308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:33.751261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:33.751223    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:33.886965 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:33.886929    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"]
Nov 02 17:46:33.891402 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:33.891383    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:33.944365 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:33.944343    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-494cff\""
Nov 02 17:46:34.020348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.020317    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-494cff-dockercfg-2t8mf\""
Nov 02 17:46:34.035923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.035879    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"]
Nov 02 17:46:34.061836 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.061798    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.062108 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.062087    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7h7\" (UniqueName: \"kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7\") pod \"payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.077084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.077051    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"]
Nov 02 17:46:34.082342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.082312    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.104001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.103972    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"]
Nov 02 17:46:34.164255 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.164168    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") " pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.164429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.164261    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6mm\" (UniqueName: \"kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm\") pod \"payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") " pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.164429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.164304    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.164429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.164357    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7h7\" (UniqueName: \"kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7\") pod \"payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.165034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.165012    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.226344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.226315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7h7\" (UniqueName: \"kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7\") pod \"payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.266205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.266167    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") " pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.266377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.266229    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6mm\" (UniqueName: \"kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm\") pod \"payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") " pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.266932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.266910    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") " pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.323591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.323559    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6mm\" (UniqueName: \"kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm\") pod \"payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") " pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.355019 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.354963    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:34.403495 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.403465    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:34.504878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.504850    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:34.800350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.800318    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"]
Nov 02 17:46:34.800987 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:46:34.800964    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e19a29c_3270_4515_ae1a_fdadd0e5541f.slice/crio-1d9f98f066376f049825ed33cd2b466af0a8355a6be36ca44ac072e78d1c1aad WatchSource:0}: Error finding container 1d9f98f066376f049825ed33cd2b466af0a8355a6be36ca44ac072e78d1c1aad: Status 404 returned error can't find the container with id 1d9f98f066376f049825ed33cd2b466af0a8355a6be36ca44ac072e78d1c1aad
Nov 02 17:46:34.906229 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:46:34.906199    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9214c37c_134a_4c60_83fc_1951688b136b.slice/crio-37d66264e1fd1cc3deb5d13943c990a04103fc22acc5dabed2797b053f7f4501 WatchSource:0}: Error finding container 37d66264e1fd1cc3deb5d13943c990a04103fc22acc5dabed2797b053f7f4501: Status 404 returned error can't find the container with id 37d66264e1fd1cc3deb5d13943c990a04103fc22acc5dabed2797b053f7f4501
Nov 02 17:46:34.906359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:34.906324    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"]
Nov 02 17:46:35.221224 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.221182    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g" event={"ID":"9214c37c-134a-4c60-83fc-1951688b136b","Type":"ContainerStarted","Data":"03b4548d0fdec4ad374c1087202ed6f444906a3e52520920e9c7c23c88404de9"}
Nov 02 17:46:35.221411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.221250    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g" event={"ID":"9214c37c-134a-4c60-83fc-1951688b136b","Type":"ContainerStarted","Data":"37d66264e1fd1cc3deb5d13943c990a04103fc22acc5dabed2797b053f7f4501"}
Nov 02 17:46:35.221411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.221373    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:35.222957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.222934    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" event={"ID":"1e19a29c-3270-4515-ae1a-fdadd0e5541f","Type":"ContainerStarted","Data":"f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143"}
Nov 02 17:46:35.223084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.222969    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" event={"ID":"1e19a29c-3270-4515-ae1a-fdadd0e5541f","Type":"ContainerStarted","Data":"1d9f98f066376f049825ed33cd2b466af0a8355a6be36ca44ac072e78d1c1aad"}
Nov 02 17:46:35.223084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.223006    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:35.339957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.339900    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g" podStartSLOduration=2.339885734 podStartE2EDuration="2.339885734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:35.337572447 +0000 UTC m=+1426.178178381" watchObservedRunningTime="2026-11-02 17:46:35.339885734 +0000 UTC m=+1426.180491671"
Nov 02 17:46:35.457464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:35.457413    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" podStartSLOduration=2.4573973860000002 podStartE2EDuration="2.457397386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:35.454042736 +0000 UTC m=+1426.294648683" watchObservedRunningTime="2026-11-02 17:46:35.457397386 +0000 UTC m=+1426.298003356"
Nov 02 17:46:36.008139 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.008105    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"]
Nov 02 17:46:36.012252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.012234    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.115673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.115641    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"distinct-2-w0-ceb1ae-kserve-self-signed-certs\""
Nov 02 17:46:36.119355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.119334    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"]
Nov 02 17:46:36.167735 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.167658    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:36.192805 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.192754    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8wz\" (UniqueName: \"kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.192999 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.192947    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.193077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.193052    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.193193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.193085    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.193193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.193185    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.193360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.193335    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294443    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294507    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294796 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294566    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8wz\" (UniqueName: \"kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294796 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294691    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294796 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294775    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294803    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294910    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.294949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.294939    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.295155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.295133    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.297337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.297309    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.297508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.297440    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.352752 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.352714    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8wz\" (UniqueName: \"kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz\") pod \"distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") " pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:36.625277 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:36.625243    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:37.009524 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:37.009495    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"]
Nov 02 17:46:37.010151 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:46:37.010125    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0018d34_408d_4cf5_8853_6b08d369fbbb.slice/crio-cfd3ba100b4767bf70a70f2365bd2c3d7879d4cd2ef420291c4f39fdad15e5c9 WatchSource:0}: Error finding container cfd3ba100b4767bf70a70f2365bd2c3d7879d4cd2ef420291c4f39fdad15e5c9: Status 404 returned error can't find the container with id cfd3ba100b4767bf70a70f2365bd2c3d7879d4cd2ef420291c4f39fdad15e5c9
Nov 02 17:46:37.238424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:37.238393    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5" event={"ID":"f0018d34-408d-4cf5-8853-6b08d369fbbb","Type":"ContainerStarted","Data":"46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742"}
Nov 02 17:46:37.238605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:37.238441    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5" event={"ID":"f0018d34-408d-4cf5-8853-6b08d369fbbb","Type":"ContainerStarted","Data":"cfd3ba100b4767bf70a70f2365bd2c3d7879d4cd2ef420291c4f39fdad15e5c9"}
Nov 02 17:46:37.359819 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:37.359775    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5" podStartSLOduration=2.359762491 podStartE2EDuration="2.359762491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:37.354945605 +0000 UTC m=+1428.195551540" watchObservedRunningTime="2026-11-02 17:46:37.359762491 +0000 UTC m=+1428.200368425"
Nov 02 17:46:37.625787 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:37.625712    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:39.421664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:39.418707    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:40.925011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:40.924970    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:41.160323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.160191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:41.319194 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.319100    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"]
Nov 02 17:46:41.328948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.328920    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.383613 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.383584    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-215cbd\""
Nov 02 17:46:41.406159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.405833    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdzp\" (UniqueName: \"kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp\") pod \"payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.406159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.405892    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.437053 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.436977    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-215cbd-dockercfg-4jk96\""
Nov 02 17:46:41.507714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.507662    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdzp\" (UniqueName: \"kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp\") pod \"payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.507962 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.507943    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.525430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.525393    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.583709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.566840    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"]
Nov 02 17:46:41.672272 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.672238    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdzp\" (UniqueName: \"kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp\") pod \"payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:41.702443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:41.702138    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:42.291209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:42.272007    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"]
Nov 02 17:46:42.309826 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:42.309790    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t" event={"ID":"74fa8a46-539d-4688-bdbf-a8dd2597d726","Type":"ContainerStarted","Data":"df0d862d8a8c3f658820808f419c3392e2bec8a1c9baf2b5f7b707e3b3a898c4"}
Nov 02 17:46:43.318699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:43.318659    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t" event={"ID":"74fa8a46-539d-4688-bdbf-a8dd2597d726","Type":"ContainerStarted","Data":"321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827"}
Nov 02 17:46:43.319555 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:43.318882    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:43.435726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:43.435663    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t" podStartSLOduration=2.435622478 podStartE2EDuration="2.435622478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:43.432998972 +0000 UTC m=+1434.273604907" watchObservedRunningTime="2026-11-02 17:46:43.435622478 +0000 UTC m=+1434.276228411"
Nov 02 17:46:44.907043 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:44.907007    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:46.232297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:46.232263    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:46:46.232726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:46.232347    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:46:46.626220 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:46.626186    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:46.639010 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:46.638987    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:47.361116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:47.361085    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 17:46:47.882176 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:47.882143    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:49.700427 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:49.700391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:54.331549 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:54.331520    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:46:54.597844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:54.597761    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:57.066230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:57.066197    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:59.058921 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:46:59.058885    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:46:59.446494 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:46:59.446464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:04.169470 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:04.169359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:08.672306 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.672270    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"]
Nov 02 17:47:08.676500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.676480    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.737394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.737361    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"]
Nov 02 17:47:08.780570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.780536    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-494cff-serving-cert\""
Nov 02 17:47:08.840944 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.840909    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.841145 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.841126    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsfd\" (UniqueName: \"kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.841263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.841235    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.841315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.841301    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.943232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.943127    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsfd\" (UniqueName: \"kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.943232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.943208    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.943484 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.943251    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.943484 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.943334    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.944131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.944102    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.946334 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.946306    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.946555 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.946535    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:08.954751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:08.954721    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:09.001458 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:09.001424    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsfd\" (UniqueName: \"kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd\") pod \"maas-api-e2e-api-b-494cff-77fdb8b6df-q882t\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:09.207564 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:09.207477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:09.290180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:09.290139    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:09.685963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:09.685932    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"]
Nov 02 17:47:09.686619 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:47:09.686596    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b1b900_6287_41ff_b663_c19037b12bdf.slice/crio-73312a01e07a64aeeffa5de6fd8df83a6c70831401e8b9a380c435cefd8807f9 WatchSource:0}: Error finding container 73312a01e07a64aeeffa5de6fd8df83a6c70831401e8b9a380c435cefd8807f9: Status 404 returned error can't find the container with id 73312a01e07a64aeeffa5de6fd8df83a6c70831401e8b9a380c435cefd8807f9
Nov 02 17:47:10.525806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:10.525768    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" event={"ID":"a6b1b900-6287-41ff-b663-c19037b12bdf","Type":"ContainerStarted","Data":"73312a01e07a64aeeffa5de6fd8df83a6c70831401e8b9a380c435cefd8807f9"}
Nov 02 17:47:11.534339 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:11.534302    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" event={"ID":"a6b1b900-6287-41ff-b663-c19037b12bdf","Type":"ContainerStarted","Data":"dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86"}
Nov 02 17:47:11.534816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:11.534377    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:11.663761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:11.663703    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" podStartSLOduration=2.649811492 podStartE2EDuration="3.663686757s" totalImagesPullingTime="1.013875265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:11.662046698 +0000 UTC m=+1462.502652631" watchObservedRunningTime="2026-11-02 17:47:11.663686757 +0000 UTC m=+1462.504292738"
Nov 02 17:47:13.042018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.041987    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"]
Nov 02 17:47:13.046252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.046234    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093072 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093028    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093340 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093311    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093346    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093516 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093467    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093560 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093550    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093584    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093701 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093618    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkwkd\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093734    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.093815 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.093758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.099092 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.099072    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ceb1ae-openshift-default-dockercfg-pmptc\""
Nov 02 17:47:13.173278 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.173242    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"]
Nov 02 17:47:13.194602 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.194565    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.194822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.194716    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.194822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.194795    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.194935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.194826    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.194935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.194911    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.195037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.194972    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.195037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.195020    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.195150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.195064    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkwkd\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.195150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.195095    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.195598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.195523    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.196039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.196014    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.196513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.196489    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.197346 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.197316    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.197774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.197581    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.198046 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.198025    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.198572 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.198554    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.256571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.256537    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.257677 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.257655    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkwkd\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.360364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.360335    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:13.807295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.807266    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"]
Nov 02 17:47:13.808833 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:47:13.808783    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9898615_30ae_4d22_86e6_0f3d190e40b0.slice/crio-d718cdb4d7dbef63683c3d8c7cc4909d29765cc2bf2936492802b9c61fe42b8f WatchSource:0}: Error finding container d718cdb4d7dbef63683c3d8c7cc4909d29765cc2bf2936492802b9c61fe42b8f: Status 404 returned error can't find the container with id d718cdb4d7dbef63683c3d8c7cc4909d29765cc2bf2936492802b9c61fe42b8f
Nov 02 17:47:13.811978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.811938    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:47:13.812075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.812057    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:47:13.812113 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:13.812103    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:47:14.167597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:14.167568    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:14.563710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:14.563024    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" event={"ID":"f9898615-30ae-4d22-86e6-0f3d190e40b0","Type":"ContainerStarted","Data":"1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9"}
Nov 02 17:47:14.563710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:14.563096    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" event={"ID":"f9898615-30ae-4d22-86e6-0f3d190e40b0","Type":"ContainerStarted","Data":"d718cdb4d7dbef63683c3d8c7cc4909d29765cc2bf2936492802b9c61fe42b8f"}
Nov 02 17:47:14.746502 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:14.744163    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podStartSLOduration=2.744142078 podStartE2EDuration="2.744142078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:14.742150181 +0000 UTC m=+1465.582756126" watchObservedRunningTime="2026-11-02 17:47:14.744142078 +0000 UTC m=+1465.584748012"
Nov 02 17:47:14.806498 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:14.806468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:15.362893 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:15.362863    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:15.364591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:15.364563    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:47:15.364743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:15.364616    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 17:47:16.363999 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:16.363957    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:47:16.364510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:16.364020    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 17:47:17.363995 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:17.363955    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:47:17.364311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:17.364279    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 17:47:17.567063 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:17.567034    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:47:18.362999 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.362962    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:47:18.363174 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.363032    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 17:47:18.575266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.575230    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"]
Nov 02 17:47:18.580380 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.580360    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.632097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.632022    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"]
Nov 02 17:47:18.675986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.675950    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-215cbd-serving-cert\""
Nov 02 17:47:18.682561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.682526    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.682893 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.682870    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.683085 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.683070    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.683175 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.683165    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt24k\" (UniqueName: \"kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.784415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.784381    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.784415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.784418    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt24k\" (UniqueName: \"kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.784644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.784523    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.784644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.784573    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.785224 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.785195    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.785764 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:47:18.785745    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-serving-cert: secret "maas-api-e2e-api-a-215cbd-serving-cert" not found
Nov 02 17:47:18.785977 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:47:18.785967    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls podName:8c3f967a-b00f-4c10-9b5f-5d5d0e077de8 nodeName:}" failed. No retries permitted until 2026-11-02 17:47:19.285945712 +0000 UTC m=+1470.126551630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls") pod "maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" (UID: "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8") : secret "maas-api-e2e-api-a-215cbd-serving-cert" not found
Nov 02 17:47:18.787382 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.787359    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.805443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.805419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:18.881399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.881358    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt24k\" (UniqueName: \"kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:18.983576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:18.983487    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:19.197276 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.197241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:19.294931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.294854    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:19.297482 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.297454    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\") pod \"maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:19.367653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.366060    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:47:19.367653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.366154    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 17:47:19.497287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.497249    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:19.913764 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:19.913731    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"]
Nov 02 17:47:19.914349 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:47:19.914319    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3f967a_b00f_4c10_9b5f_5d5d0e077de8.slice/crio-81a2d9633bbac1e589188d0ae97ad8032ad80f852179e7ede2a783d1af4c0d86 WatchSource:0}: Error finding container 81a2d9633bbac1e589188d0ae97ad8032ad80f852179e7ede2a783d1af4c0d86: Status 404 returned error can't find the container with id 81a2d9633bbac1e589188d0ae97ad8032ad80f852179e7ede2a783d1af4c0d86
Nov 02 17:47:20.361184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:20.361150    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:47:20.361345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:20.361229    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 17:47:20.616527 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:20.616445    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" event={"ID":"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8","Type":"ContainerStarted","Data":"81a2d9633bbac1e589188d0ae97ad8032ad80f852179e7ede2a783d1af4c0d86"}
Nov 02 17:47:21.366029 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:21.366003    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:21.366472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:21.366453    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:21.367119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:21.367098    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:47:21.625313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:21.625225    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" event={"ID":"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8","Type":"ContainerStarted","Data":"7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767"}
Nov 02 17:47:21.625313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:21.625284    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:21.797358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:21.797299    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" podStartSLOduration=2.908503769 podStartE2EDuration="3.797284922s" totalImagesPullingTime="888.781153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:21.793912215 +0000 UTC m=+1472.634518149" watchObservedRunningTime="2026-11-02 17:47:21.797284922 +0000 UTC m=+1472.637890895"
Nov 02 17:47:23.861830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:23.861757    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:27.637233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:27.637206    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:47:28.774740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:28.774705    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:29.062079 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:47:29.062000    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:47:33.482368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:33.482276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:38.269171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:38.269130    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:42.038395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:42.038345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:43.221958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:43.221909    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:43.483254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:43.483153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:43.705001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:43.704961    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:48.163871 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:48.163810    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:48.352214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:48.352175    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:48.622116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:48.622073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:49.580245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:49.580217    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:47:49.586476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:49.586449    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:47:51.358344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:51.358310    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:51.559349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:51.559311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:54.586581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:54.586496    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:55.227575 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:55.227536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:55.418122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:55.418090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:59.060734 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:47:59.060692    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:47:59.973266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:47:59.973234    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:05.064308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:05.064272    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:07.126897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:07.126861    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:08.906727 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:08.906696    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:10.251042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:10.251012    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:14.766387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:14.766349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:15.163438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:15.163406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:18.229826 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:18.229789    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:18.624603 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:18.624566    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"]
Nov 02 17:48:18.730854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:18.730816    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"]
Nov 02 17:48:18.982504 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:18.982399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:19.190042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:19.189985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:20.009150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:20.009065    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:20.269914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:20.269829    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:21.231655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:21.228193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:22.022647 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:22.022120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:22.222966 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:22.222929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:23.625709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:23.625660    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r" podUID="f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" containerName="payload-processing" containerID="cri-o://6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289" gracePeriod=25
Nov 02 17:48:23.732034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:23.731999    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb" podUID="b325e916-7c24-4814-ab7b-1287b7b8e524" containerName="payload-pre-processing" containerID="cri-o://594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d" gracePeriod=25
Nov 02 17:48:23.958554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:23.958527    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:48:24.042212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.042189    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:48:24.088396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.088352    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume\") pod \"b325e916-7c24-4814-ab7b-1287b7b8e524\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") "
Nov 02 17:48:24.088576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.088433    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68\" (UniqueName: \"kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68\") pod \"b325e916-7c24-4814-ab7b-1287b7b8e524\" (UID: \"b325e916-7c24-4814-ab7b-1287b7b8e524\") "
Nov 02 17:48:24.088576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.088503    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume\") pod \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") "
Nov 02 17:48:24.088576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.088543    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7\" (UniqueName: \"kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7\") pod \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\" (UID: \"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67\") "
Nov 02 17:48:24.089957 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.089930    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume" pod "b325e916-7c24-4814-ab7b-1287b7b8e524" (UID: "b325e916-7c24-4814-ab7b-1287b7b8e524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:24.090115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.089984    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume" pod "f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" (UID: "f0bd3d9e-b69d-49f1-8596-2ae96ae13d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:24.092012 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.091966    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68" pod "b325e916-7c24-4814-ab7b-1287b7b8e524" (UID: "b325e916-7c24-4814-ab7b-1287b7b8e524"). InnerVolumeSpecName "kube-api-access-cpw68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:24.092139 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.092079    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7" pod "f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" (UID: "f0bd3d9e-b69d-49f1-8596-2ae96ae13d67"). InnerVolumeSpecName "kube-api-access-8grz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:24.190725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.190652    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpw68\" (UniqueName: \"kubernetes.io/projected/b325e916-7c24-4814-ab7b-1287b7b8e524-kube-api-access-cpw68\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:24.190725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.190678    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:24.190725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.190688    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grz7\" (UniqueName: \"kubernetes.io/projected/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67-kube-api-access-8grz7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:24.190725 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.190696    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b325e916-7c24-4814-ab7b-1287b7b8e524-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:24.211193 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.211160    4476 generic.go:356] "Generic (PLEG): container finished" podID="f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" containerID="6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289" exitCode=0
Nov 02 17:48:24.211373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.211213    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"
Nov 02 17:48:24.211373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.211256    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r" event={"ID":"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67","Type":"ContainerDied","Data":"6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289"}
Nov 02 17:48:24.211373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.211324    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r" event={"ID":"f0bd3d9e-b69d-49f1-8596-2ae96ae13d67","Type":"ContainerDied","Data":"ddb7d62f154bb8ce778ac4221ff85a2af2e87ab79ad570d04a44973fc3fe622e"}
Nov 02 17:48:24.211373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.211355    4476 scope.go:122] "RemoveContainer" containerID="6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289"
Nov 02 17:48:24.213001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.212886    4476 generic.go:356] "Generic (PLEG): container finished" podID="b325e916-7c24-4814-ab7b-1287b7b8e524" containerID="594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d" exitCode=0
Nov 02 17:48:24.213001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.212923    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb" event={"ID":"b325e916-7c24-4814-ab7b-1287b7b8e524","Type":"ContainerDied","Data":"594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d"}
Nov 02 17:48:24.213001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.212946    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"
Nov 02 17:48:24.213001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.212964    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb" event={"ID":"b325e916-7c24-4814-ab7b-1287b7b8e524","Type":"ContainerDied","Data":"1814e1bffa1b66ed0cd6dd7b6aec06e7efcd6bbf474c390b98e9bea9be22a443"}
Nov 02 17:48:24.222341 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.222282    4476 scope.go:122] "RemoveContainer" containerID="6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289"
Nov 02 17:48:24.222691 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:24.222658    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289\": container with ID starting with 6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289 not found: ID does not exist" containerID="6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289"
Nov 02 17:48:24.222762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.222692    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289"} err="failed to get container status \"6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289\": rpc error: code = NotFound desc = could not find container \"6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289\": container with ID starting with 6f6adaea3076b6beadc6ce4b4305a723741e3d976b4c12faadec1fd369ef5289 not found: ID does not exist"
Nov 02 17:48:24.222762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.222713    4476 scope.go:122] "RemoveContainer" containerID="594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d"
Nov 02 17:48:24.234445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.234421    4476 scope.go:122] "RemoveContainer" containerID="594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d"
Nov 02 17:48:24.234800 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:24.234769    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d\": container with ID starting with 594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d not found: ID does not exist" containerID="594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d"
Nov 02 17:48:24.234887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.234812    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d"} err="failed to get container status \"594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d\": rpc error: code = NotFound desc = could not find container \"594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d\": container with ID starting with 594c17e392d98e629acfa36e39ee93eec0cc80bb2c56df90f1518c35cef0bb5d not found: ID does not exist"
Nov 02 17:48:24.377653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.377026    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"]
Nov 02 17:48:24.381488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.381463    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8d61e-765958ff75-sphkb"]
Nov 02 17:48:24.489552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.489454    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"]
Nov 02 17:48:24.493931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:24.493907    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8d61e-b45cfcd87-6pv6r"]
Nov 02 17:48:25.384595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:25.384560    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b325e916-7c24-4814-ab7b-1287b7b8e524" path="/var/lib/kubelet/pods/b325e916-7c24-4814-ab7b-1287b7b8e524/volumes"
Nov 02 17:48:25.385074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:25.385057    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" path="/var/lib/kubelet/pods/f0bd3d9e-b69d-49f1-8596-2ae96ae13d67/volumes"
Nov 02 17:48:26.709073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:26.709036    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:26.914331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:26.914294    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:27.118239 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:27.118207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:27.337762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:27.337731    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:27.535498 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:27.535398    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:27.833349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:27.833259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:29.059424 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:29.059389    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:48:31.655308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:31.655275    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:31.884162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:31.884109    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:32.004122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004040    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"]
Nov 02 17:48:32.004615 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004600    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b325e916-7c24-4814-ab7b-1287b7b8e524" containerName="payload-pre-processing"
Nov 02 17:48:32.004721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004616    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b325e916-7c24-4814-ab7b-1287b7b8e524" containerName="payload-pre-processing"
Nov 02 17:48:32.004721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004652    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" containerName="payload-processing"
Nov 02 17:48:32.004721 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004659    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" containerName="payload-processing"
Nov 02 17:48:32.004891 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004773    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b325e916-7c24-4814-ab7b-1287b7b8e524" containerName="payload-pre-processing"
Nov 02 17:48:32.004891 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.004783    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bd3d9e-b69d-49f1-8596-2ae96ae13d67" containerName="payload-processing"
Nov 02 17:48:32.009666 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.009648    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.043528 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.043490    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.043784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.043566    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn948\" (UniqueName: \"kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948\") pod \"payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.067872 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.067839    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"]
Nov 02 17:48:32.112417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.112382    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f0f3e1-dockercfg-jccjm\""
Nov 02 17:48:32.112614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.112557    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f0f3e1\""
Nov 02 17:48:32.117202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.117176    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"]
Nov 02 17:48:32.117427 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.117405    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" podUID="c6eeef77-9a60-47e0-8d06-977ca9764adf" containerName="maas-api" containerID="cri-o://52e6e4a4aaad39b6ddd2f80ac99372be05a13299fefd35a2f6f7c975e8cc5eb5" gracePeriod=30
Nov 02 17:48:32.145622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.144738    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.145790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.145577    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.145870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.145850    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn948\" (UniqueName: \"kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948\") pod \"payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.160214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.160179    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"]
Nov 02 17:48:32.165613 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.165585    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.204574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.204543    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn948\" (UniqueName: \"kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948\") pod \"payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.250650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.249976    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") " pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.250650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.250049    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qml9r\" (UniqueName: \"kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r\") pod \"payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") " pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.267906 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.267847    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"]
Nov 02 17:48:32.320901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.320768    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:32.322869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.322839    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:32.330901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.329887    4476 generic.go:356] "Generic (PLEG): container finished" podID="c6eeef77-9a60-47e0-8d06-977ca9764adf" containerID="52e6e4a4aaad39b6ddd2f80ac99372be05a13299fefd35a2f6f7c975e8cc5eb5" exitCode=0
Nov 02 17:48:32.330901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.330142    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" event={"ID":"c6eeef77-9a60-47e0-8d06-977ca9764adf","Type":"ContainerDied","Data":"52e6e4a4aaad39b6ddd2f80ac99372be05a13299fefd35a2f6f7c975e8cc5eb5"}
Nov 02 17:48:32.352284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.351616    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") " pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.352284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.351721    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qml9r\" (UniqueName: \"kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r\") pod \"payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") " pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.354118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.354084    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") " pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.522778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.522664    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qml9r\" (UniqueName: \"kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r\") pod \"payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") " pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.603756 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.579882    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.660668    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.665709    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs\") pod \"c6eeef77-9a60-47e0-8d06-977ca9764adf\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") "
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.665772    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\") pod \"c6eeef77-9a60-47e0-8d06-977ca9764adf\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") "
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.665810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca\") pod \"c6eeef77-9a60-47e0-8d06-977ca9764adf\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") "
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.665974    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht\" (UniqueName: \"kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht\") pod \"c6eeef77-9a60-47e0-8d06-977ca9764adf\" (UID: \"c6eeef77-9a60-47e0-8d06-977ca9764adf\") "
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.667811    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca" pod "c6eeef77-9a60-47e0-8d06-977ca9764adf" (UID: "c6eeef77-9a60-47e0-8d06-977ca9764adf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:32.678399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.675046    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls" pod "c6eeef77-9a60-47e0-8d06-977ca9764adf" (UID: "c6eeef77-9a60-47e0-8d06-977ca9764adf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:48:32.680481 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.678973    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs" pod "c6eeef77-9a60-47e0-8d06-977ca9764adf" (UID: "c6eeef77-9a60-47e0-8d06-977ca9764adf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:48:32.734671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.731686    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht" pod "c6eeef77-9a60-47e0-8d06-977ca9764adf" (UID: "c6eeef77-9a60-47e0-8d06-977ca9764adf"). InnerVolumeSpecName "kube-api-access-z9cht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:32.775124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.768960    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9cht\" (UniqueName: \"kubernetes.io/projected/c6eeef77-9a60-47e0-8d06-977ca9764adf-kube-api-access-z9cht\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:32.775124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.769000    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:32.775124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.769017    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6eeef77-9a60-47e0-8d06-977ca9764adf-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:32.775124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.769030    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eeef77-9a60-47e0-8d06-977ca9764adf-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:32.902567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:32.902527    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"]
Nov 02 17:48:32.907206 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:48:32.907166    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994ecfe6_ca82_4dae_8ebf_e8c4675c9af0.slice/crio-d10f8efaf4e415cf684c02efcba861ee850a09b53449bffbc5f4349998d0e457 WatchSource:0}: Error finding container d10f8efaf4e415cf684c02efcba861ee850a09b53449bffbc5f4349998d0e457: Status 404 returned error can't find the container with id d10f8efaf4e415cf684c02efcba861ee850a09b53449bffbc5f4349998d0e457
Nov 02 17:48:33.130157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.130130    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"]
Nov 02 17:48:33.132439 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:48:33.132379    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc811840_7d78_4c9d_b29f_927c1c411944.slice/crio-64fe906f505f94b4264d60e6e0e665f02f3b86e663b3584e47ba64ecb80a006e WatchSource:0}: Error finding container 64fe906f505f94b4264d60e6e0e665f02f3b86e663b3584e47ba64ecb80a006e: Status 404 returned error can't find the container with id 64fe906f505f94b4264d60e6e0e665f02f3b86e663b3584e47ba64ecb80a006e
Nov 02 17:48:33.340368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.340326    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6" event={"ID":"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0","Type":"ContainerStarted","Data":"d857f1c15b05427ae83c460c6c23b676ee141f4dd3db8f8c4c1ae8e3c08fe0bf"}
Nov 02 17:48:33.340368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.340396    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6" event={"ID":"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0","Type":"ContainerStarted","Data":"d10f8efaf4e415cf684c02efcba861ee850a09b53449bffbc5f4349998d0e457"}
Nov 02 17:48:33.340368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.340453    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:33.342843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.342808    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n" event={"ID":"c6eeef77-9a60-47e0-8d06-977ca9764adf","Type":"ContainerDied","Data":"2fed0c69f3dec6ec9ea17a53c1913bf70c206d800bd010da392a269a86e0bf05"}
Nov 02 17:48:33.342979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.342872    4476 scope.go:122] "RemoveContainer" containerID="52e6e4a4aaad39b6ddd2f80ac99372be05a13299fefd35a2f6f7c975e8cc5eb5"
Nov 02 17:48:33.343075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.343061    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"
Nov 02 17:48:33.369698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.357712    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm" event={"ID":"fc811840-7d78-4c9d-b29f-927c1c411944","Type":"ContainerStarted","Data":"89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b"}
Nov 02 17:48:33.369698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.357778    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm" event={"ID":"fc811840-7d78-4c9d-b29f-927c1c411944","Type":"ContainerStarted","Data":"64fe906f505f94b4264d60e6e0e665f02f3b86e663b3584e47ba64ecb80a006e"}
Nov 02 17:48:33.369698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.358535    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:33.400349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.400313    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:33.455738 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.455675    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6" podStartSLOduration=2.455660553 podStartE2EDuration="2.455660553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:33.454011926 +0000 UTC m=+1544.294617859" watchObservedRunningTime="2026-11-02 17:48:33.455660553 +0000 UTC m=+1544.296266482"
Nov 02 17:48:33.571161 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.571123    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"]
Nov 02 17:48:33.592234 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.592199    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8d61e-7f6d57d686-w8g8n"]
Nov 02 17:48:33.641374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.641287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:33.693298 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:33.693235    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm" podStartSLOduration=1.693214179 podStartE2EDuration="1.693214179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:33.687951421 +0000 UTC m=+1544.528557333" watchObservedRunningTime="2026-11-02 17:48:33.693214179 +0000 UTC m=+1544.533820116"
Nov 02 17:48:34.353343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.353308    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"]
Nov 02 17:48:34.354390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.354365    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c6eeef77-9a60-47e0-8d06-977ca9764adf" containerName="maas-api"
Nov 02 17:48:34.354390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.354388    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eeef77-9a60-47e0-8d06-977ca9764adf" containerName="maas-api"
Nov 02 17:48:34.354617 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.354549    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eeef77-9a60-47e0-8d06-977ca9764adf" containerName="maas-api"
Nov 02 17:48:34.360183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.360150    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.412128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.412080    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e61288-serving-cert\""
Nov 02 17:48:34.414948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.414919    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"]
Nov 02 17:48:34.508055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.501484    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.508055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.502126    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdwg4\" (UniqueName: \"kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.508055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.502174    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.508055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.502242    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.603460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.603356    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.603460 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:34.603546    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-serving-cert: secret "maas-api-e2e-api-a-e61288-serving-cert" not found
Nov 02 17:48:34.605073 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:34.604825    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls podName:9260d9ed-3ae3-4562-84a8-35151a37bd62 nodeName:}" failed. No retries permitted until 2026-11-02 17:48:35.104798223 +0000 UTC m=+1545.945404159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls") pod "maas-api-e2e-api-a-e61288-545b69c84-v666v" (UID: "9260d9ed-3ae3-4562-84a8-35151a37bd62") : secret "maas-api-e2e-api-a-e61288-serving-cert" not found
Nov 02 17:48:34.606256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.603599    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdwg4\" (UniqueName: \"kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.606256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.605447    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.606256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.605496    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.606656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.606605    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.620132 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.620105    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.621641 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.621599    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:34.666468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.666432    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdwg4\" (UniqueName: \"kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:34.825533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:34.825493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:35.112110 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:35.112067    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:35.114641 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:35.114611    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\") pod \"maas-api-e2e-api-a-e61288-545b69c84-v666v\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:35.278188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:35.278150    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:35.396267 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:35.390942    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6eeef77-9a60-47e0-8d06-977ca9764adf" path="/var/lib/kubelet/pods/c6eeef77-9a60-47e0-8d06-977ca9764adf/volumes"
Nov 02 17:48:35.658513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:35.658476    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"]
Nov 02 17:48:35.662319 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:48:35.662290    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9260d9ed_3ae3_4562_84a8_35151a37bd62.slice/crio-ef61677e8d4734aad3c8f4ac156e8c0478e64c1f97b1451e8db05158c0eb111c WatchSource:0}: Error finding container ef61677e8d4734aad3c8f4ac156e8c0478e64c1f97b1451e8db05158c0eb111c: Status 404 returned error can't find the container with id ef61677e8d4734aad3c8f4ac156e8c0478e64c1f97b1451e8db05158c0eb111c
Nov 02 17:48:36.390803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:36.390768    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v" event={"ID":"9260d9ed-3ae3-4562-84a8-35151a37bd62","Type":"ContainerStarted","Data":"ef61677e8d4734aad3c8f4ac156e8c0478e64c1f97b1451e8db05158c0eb111c"}
Nov 02 17:48:37.399871 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:37.399820    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v" event={"ID":"9260d9ed-3ae3-4562-84a8-35151a37bd62","Type":"ContainerStarted","Data":"fce3534e611bc3dd0c30799b2c006194980243f962f82d14c46001184dfcbc35"}
Nov 02 17:48:37.400360 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:37.399924    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:37.515612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:37.515560    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v" podStartSLOduration=2.572197642 podStartE2EDuration="3.515542973s" totalImagesPullingTime="943.345331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:37.513252074 +0000 UTC m=+1548.353858010" watchObservedRunningTime="2026-11-02 17:48:37.515542973 +0000 UTC m=+1548.356148908"
Nov 02 17:48:38.896413 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:38.896377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:39.096246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.096207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:39.264560 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.264464    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"]
Nov 02 17:48:39.268932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.268906    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.328622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.328579    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"]
Nov 02 17:48:39.387689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.385902    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fa615-dockercfg-cbknf\""
Nov 02 17:48:39.387689 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.386024    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fa615\""
Nov 02 17:48:39.391613 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.391574    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.391776 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.391679    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstr7\" (UniqueName: \"kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.439519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.439485    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"]
Nov 02 17:48:39.443959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.443885    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.467584 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.467352    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"]
Nov 02 17:48:39.494222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.494181    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstr7\" (UniqueName: \"kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.494429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.494351    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.494533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.494512    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksrb\" (UniqueName: \"kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb\") pod \"payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.494694 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.494672    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.495749 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.495721    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.555204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.555128    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstr7\" (UniqueName: \"kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.593978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.593943    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:39.596863 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.596829    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.597044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.596981    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksrb\" (UniqueName: \"kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb\") pod \"payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.597608 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.597588    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.656131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.656077    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksrb\" (UniqueName: \"kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb\") pod \"payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.770058 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.770023    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:39.987758 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:48:39.987726    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a16278d_c430_45c2_9118_7f22ce22b117.slice/crio-b92af612016d5700e119dc5ad275db92184449453a0bc1b38aaf9b049023b0e7 WatchSource:0}: Error finding container b92af612016d5700e119dc5ad275db92184449453a0bc1b38aaf9b049023b0e7: Status 404 returned error can't find the container with id b92af612016d5700e119dc5ad275db92184449453a0bc1b38aaf9b049023b0e7
Nov 02 17:48:39.988514 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:39.988489    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"]
Nov 02 17:48:40.155518 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:48:40.155478    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7dcd12_61aa_4d4f_986e_95cbd703b15a.slice/crio-faabd02763411c6115c888fbb57fa6598ec37a2fc6492eb76d9a4bf2e222e77d WatchSource:0}: Error finding container faabd02763411c6115c888fbb57fa6598ec37a2fc6492eb76d9a4bf2e222e77d: Status 404 returned error can't find the container with id faabd02763411c6115c888fbb57fa6598ec37a2fc6492eb76d9a4bf2e222e77d
Nov 02 17:48:40.155854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.155834    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"]
Nov 02 17:48:40.436788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.436691    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w" event={"ID":"9a16278d-c430-45c2-9118-7f22ce22b117","Type":"ContainerStarted","Data":"fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a"}
Nov 02 17:48:40.436788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.436752    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w" event={"ID":"9a16278d-c430-45c2-9118-7f22ce22b117","Type":"ContainerStarted","Data":"b92af612016d5700e119dc5ad275db92184449453a0bc1b38aaf9b049023b0e7"}
Nov 02 17:48:40.436788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.436788    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:40.438432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.438401    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr" event={"ID":"9a7dcd12-61aa-4d4f-986e-95cbd703b15a","Type":"ContainerStarted","Data":"5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c"}
Nov 02 17:48:40.438579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.438451    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:40.438579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.438466    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr" event={"ID":"9a7dcd12-61aa-4d4f-986e-95cbd703b15a","Type":"ContainerStarted","Data":"faabd02763411c6115c888fbb57fa6598ec37a2fc6492eb76d9a4bf2e222e77d"}
Nov 02 17:48:40.490480 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.490440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:40.557134 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.557065    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w" podStartSLOduration=1.557045829 podStartE2EDuration="1.557045829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:40.553507972 +0000 UTC m=+1551.394113906" watchObservedRunningTime="2026-11-02 17:48:40.557045829 +0000 UTC m=+1551.397651763"
Nov 02 17:48:40.674941 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:40.668947    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr" podStartSLOduration=1.668924834 podStartE2EDuration="1.668924834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:40.667077489 +0000 UTC m=+1551.507683424" watchObservedRunningTime="2026-11-02 17:48:40.668924834 +0000 UTC m=+1551.509530769"
Nov 02 17:48:41.792464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:41.792429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:41.987463 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:41.987429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:43.412890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:43.412863    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:48:43.445443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:43.445405    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:43.948707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:43.948669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:44.374307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:44.374282    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:48:44.746221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:44.746138    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:45.384958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:45.384923    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:48:45.593686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:45.593659    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:48.544394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:48.544362    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:50.434456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:50.434412    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:50.617323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:50.617292    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:51.194288 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:51.194256    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"]
Nov 02 17:48:51.306162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:51.306128    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"]
Nov 02 17:48:51.451078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:51.450999    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:48:51.451431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:51.451084    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:48:51.609116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:51.609068    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:51.805695 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:51.805584    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:55.778068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:55.778032    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:55.985971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:55.985926    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:56.195485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.195444    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" podUID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" containerName="payload-processing" containerID="cri-o://f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143" gracePeriod=25
Nov 02 17:48:56.230805 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:48:56.230780    4476 logging.go:55] [core] [Channel #1548 SubChannel #1550]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.92:50052", ServerName: "10.133.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.92:50052: connect: connection refused"
Nov 02 17:48:56.307097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.307057    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g" podUID="9214c37c-134a-4c60-83fc-1951688b136b" containerName="payload-pre-processing" containerID="cri-o://03b4548d0fdec4ad374c1087202ed6f444906a3e52520920e9c7c23c88404de9" gracePeriod=25
Nov 02 17:48:56.502822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.502793    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:48:56.538854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.538821    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume\") pod \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") "
Nov 02 17:48:56.539328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.538931    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm\" (UniqueName: \"kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm\") pod \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\" (UID: \"1e19a29c-3270-4515-ae1a-fdadd0e5541f\") "
Nov 02 17:48:56.540345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.540293    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume" pod "1e19a29c-3270-4515-ae1a-fdadd0e5541f" (UID: "1e19a29c-3270-4515-ae1a-fdadd0e5541f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:56.544279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.544240    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm" pod "1e19a29c-3270-4515-ae1a-fdadd0e5541f" (UID: "1e19a29c-3270-4515-ae1a-fdadd0e5541f"). InnerVolumeSpecName "kube-api-access-st6mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:56.588803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.588250    4476 generic.go:356] "Generic (PLEG): container finished" podID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" containerID="f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143" exitCode=0
Nov 02 17:48:56.588803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.588489    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" event={"ID":"1e19a29c-3270-4515-ae1a-fdadd0e5541f","Type":"ContainerDied","Data":"f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143"}
Nov 02 17:48:56.588803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.588585    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" event={"ID":"1e19a29c-3270-4515-ae1a-fdadd0e5541f","Type":"ContainerDied","Data":"1d9f98f066376f049825ed33cd2b466af0a8355a6be36ca44ac072e78d1c1aad"}
Nov 02 17:48:56.588803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.588653    4476 scope.go:122] "RemoveContainer" containerID="f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143"
Nov 02 17:48:56.589126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.588866    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"
Nov 02 17:48:56.597936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.597894    4476 generic.go:356] "Generic (PLEG): container finished" podID="9214c37c-134a-4c60-83fc-1951688b136b" containerID="03b4548d0fdec4ad374c1087202ed6f444906a3e52520920e9c7c23c88404de9" exitCode=0
Nov 02 17:48:56.598067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.597940    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g" event={"ID":"9214c37c-134a-4c60-83fc-1951688b136b","Type":"ContainerDied","Data":"03b4548d0fdec4ad374c1087202ed6f444906a3e52520920e9c7c23c88404de9"}
Nov 02 17:48:56.612538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.612515    4476 scope.go:122] "RemoveContainer" containerID="f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143"
Nov 02 17:48:56.612831 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:56.612811    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143\": container with ID starting with f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143 not found: ID does not exist" containerID="f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143"
Nov 02 17:48:56.612900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.612844    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143"} err="failed to get container status \"f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143\": rpc error: code = NotFound desc = could not find container \"f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143\": container with ID starting with f749709bf93b3c2f9b9f7b1a276233cac8f545c96623a93f137b5cf82d142143 not found: ID does not exist"
Nov 02 17:48:56.613151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.613135    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:48:56.638597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.638566    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:56.640736 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.640711    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7\" (UniqueName: \"kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7\") pod \"9214c37c-134a-4c60-83fc-1951688b136b\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") "
Nov 02 17:48:56.641057 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.641036    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume\") pod \"9214c37c-134a-4c60-83fc-1951688b136b\" (UID: \"9214c37c-134a-4c60-83fc-1951688b136b\") "
Nov 02 17:48:56.641486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.641464    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e19a29c-3270-4515-ae1a-fdadd0e5541f-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.641486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.641485    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6mm\" (UniqueName: \"kubernetes.io/projected/1e19a29c-3270-4515-ae1a-fdadd0e5541f-kube-api-access-st6mm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.641847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.641813    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume" pod "9214c37c-134a-4c60-83fc-1951688b136b" (UID: "9214c37c-134a-4c60-83fc-1951688b136b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:56.644173 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.644147    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7" pod "9214c37c-134a-4c60-83fc-1951688b136b" (UID: "9214c37c-134a-4c60-83fc-1951688b136b"). InnerVolumeSpecName "kube-api-access-vl7h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:56.742806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.742716    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214c37c-134a-4c60-83fc-1951688b136b-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.742806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.742750    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7h7\" (UniqueName: \"kubernetes.io/projected/9214c37c-134a-4c60-83fc-1951688b136b-kube-api-access-vl7h7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.833166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.833135    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:56.851058 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.851017    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"]
Nov 02 17:48:56.854902 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:56.854876    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v"]
Nov 02 17:48:57.230823 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.230778    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-494cff-6bd565b85c-cc98v" podUID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.92:50052\" within 1s: context deadline exceeded"
Nov 02 17:48:57.387819 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.387789    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" path="/var/lib/kubelet/pods/1e19a29c-3270-4515-ae1a-fdadd0e5541f/volumes"
Nov 02 17:48:57.607419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.607347    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"
Nov 02 17:48:57.607419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.607346    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g" event={"ID":"9214c37c-134a-4c60-83fc-1951688b136b","Type":"ContainerDied","Data":"37d66264e1fd1cc3deb5d13943c990a04103fc22acc5dabed2797b053f7f4501"}
Nov 02 17:48:57.607419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.607416    4476 scope.go:122] "RemoveContainer" containerID="03b4548d0fdec4ad374c1087202ed6f444906a3e52520920e9c7c23c88404de9"
Nov 02 17:48:57.775845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.775814    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"]
Nov 02 17:48:57.780765 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:57.780747    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-494cff-69cc6ff599-fl65g"]
Nov 02 17:48:59.060358 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:48:59.060310    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:48:59.385909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:48:59.385874    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9214c37c-134a-4c60-83fc-1951688b136b" path="/var/lib/kubelet/pods/9214c37c-134a-4c60-83fc-1951688b136b/volumes"
Nov 02 17:49:00.875479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:00.875388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:01.873486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:01.873446    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:05.695489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:05.695432    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:10.390372 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:10.390316    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:14.490400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:14.490369    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:16.024052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:16.024020    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:16.266016 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:16.265981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:16.466680 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:16.466620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:16.666390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:16.666356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:20.505539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:20.505497    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:21.702377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:21.702342    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:24.125460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:24.125423    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:24.308816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:24.308784    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:26.422821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:26.422781    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:26.611109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:26.611071    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:27.027413 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.027375    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:27.112309 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.112265    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"]
Nov 02 17:49:27.113021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.112996    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9214c37c-134a-4c60-83fc-1951688b136b" containerName="payload-pre-processing"
Nov 02 17:49:27.113318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.113288    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9214c37c-134a-4c60-83fc-1951688b136b" containerName="payload-pre-processing"
Nov 02 17:49:27.113460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.113344    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" containerName="payload-processing"
Nov 02 17:49:27.113460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.113361    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" containerName="payload-processing"
Nov 02 17:49:27.113567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.113502    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9214c37c-134a-4c60-83fc-1951688b136b" containerName="payload-pre-processing"
Nov 02 17:49:27.113567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.113514    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e19a29c-3270-4515-ae1a-fdadd0e5541f" containerName="payload-processing"
Nov 02 17:49:27.118073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.118052    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.154353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.154317    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhgc\" (UniqueName: \"kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.154353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.154364    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.173907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.173873    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"]
Nov 02 17:49:27.217249 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.217212    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8cbedb9f-dockercfg-hxsc6\""
Nov 02 17:49:27.217451 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.217374    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8cbedb9f\""
Nov 02 17:49:27.262314 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.260694    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhgc\" (UniqueName: \"kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.262608 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.262577    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.264559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.264527    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.265456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.265427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:27.320735 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.320487    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhgc\" (UniqueName: \"kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc\") pod \"payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.433323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.433284    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:27.816772 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.816744    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"]
Nov 02 17:49:27.818063 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:49:27.818004    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16c95fe_e82f_480e_a274_ebf438f12ab0.slice/crio-5d4d6e00145b84314aa9d00e2d2d63641e3ce7f30eb729c22844ea4462be5ae3 WatchSource:0}: Error finding container 5d4d6e00145b84314aa9d00e2d2d63641e3ce7f30eb729c22844ea4462be5ae3: Status 404 returned error can't find the container with id 5d4d6e00145b84314aa9d00e2d2d63641e3ce7f30eb729c22844ea4462be5ae3
Nov 02 17:49:27.913579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.913535    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p" event={"ID":"b16c95fe-e82f-480e-a274-ebf438f12ab0","Type":"ContainerStarted","Data":"da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7"}
Nov 02 17:49:27.913763 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.913605    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p" event={"ID":"b16c95fe-e82f-480e-a274-ebf438f12ab0","Type":"ContainerStarted","Data":"5d4d6e00145b84314aa9d00e2d2d63641e3ce7f30eb729c22844ea4462be5ae3"}
Nov 02 17:49:27.913820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:27.913767    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:28.035653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:28.029806    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p" podStartSLOduration=1.029785449 podStartE2EDuration="1.029785449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:28.027719878 +0000 UTC m=+1598.868325813" watchObservedRunningTime="2026-11-02 17:49:28.029785449 +0000 UTC m=+1598.870391384"
Nov 02 17:49:29.086582 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:49:29.086539    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:49:34.803024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:34.802993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:37.422099 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:37.422056    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:38.924119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:38.924087    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:49:40.456246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:40.456211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:41.172104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:41.172073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:41.881017 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:41.880985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:42.233400 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:42.233304    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:43.315132 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:43.315096    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:47.207009 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.206976    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:47.485657 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.485555    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:47.488857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.488835    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"]
Nov 02 17:49:47.489052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.489032    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" podUID="a6b1b900-6287-41ff-b663-c19037b12bdf" containerName="maas-api" containerID="cri-o://dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86" gracePeriod=30
Nov 02 17:49:47.541778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.541585    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" podUID="a6b1b900-6287-41ff-b663-c19037b12bdf" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.95:8443/health\": dial tcp 10.133.0.95:8443: connect: connection refused"
Nov 02 17:49:47.857751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.853795    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:47.954028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:47.953998    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:49:48.005138 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.005103    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca\") pod \"a6b1b900-6287-41ff-b663-c19037b12bdf\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") "
Nov 02 17:49:48.005836 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.005766    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs\") pod \"a6b1b900-6287-41ff-b663-c19037b12bdf\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") "
Nov 02 17:49:48.006474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.005703    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca" pod "a6b1b900-6287-41ff-b663-c19037b12bdf" (UID: "a6b1b900-6287-41ff-b663-c19037b12bdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:48.006474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.006251    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd\" (UniqueName: \"kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd\") pod \"a6b1b900-6287-41ff-b663-c19037b12bdf\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") "
Nov 02 17:49:48.006474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.006342    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls\") pod \"a6b1b900-6287-41ff-b663-c19037b12bdf\" (UID: \"a6b1b900-6287-41ff-b663-c19037b12bdf\") "
Nov 02 17:49:48.007605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.006945    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b1b900-6287-41ff-b663-c19037b12bdf-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:48.011369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.011342    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs" pod "a6b1b900-6287-41ff-b663-c19037b12bdf" (UID: "a6b1b900-6287-41ff-b663-c19037b12bdf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:49:48.011726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.011704    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd" pod "a6b1b900-6287-41ff-b663-c19037b12bdf" (UID: "a6b1b900-6287-41ff-b663-c19037b12bdf"). InnerVolumeSpecName "kube-api-access-pcsfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:48.014847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.014822    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls" pod "a6b1b900-6287-41ff-b663-c19037b12bdf" (UID: "a6b1b900-6287-41ff-b663-c19037b12bdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:49:48.074539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.074501    4476 generic.go:356] "Generic (PLEG): container finished" podID="a6b1b900-6287-41ff-b663-c19037b12bdf" containerID="dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86" exitCode=0
Nov 02 17:49:48.074739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.074553    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"
Nov 02 17:49:48.074739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.074584    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" event={"ID":"a6b1b900-6287-41ff-b663-c19037b12bdf","Type":"ContainerDied","Data":"dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86"}
Nov 02 17:49:48.074739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.074662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t" event={"ID":"a6b1b900-6287-41ff-b663-c19037b12bdf","Type":"ContainerDied","Data":"73312a01e07a64aeeffa5de6fd8df83a6c70831401e8b9a380c435cefd8807f9"}
Nov 02 17:49:48.074739 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.074693    4476 scope.go:122] "RemoveContainer" containerID="dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86"
Nov 02 17:49:48.091013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.090995    4476 scope.go:122] "RemoveContainer" containerID="dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86"
Nov 02 17:49:48.091381 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:49:48.091359    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86\": container with ID starting with dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86 not found: ID does not exist" containerID="dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86"
Nov 02 17:49:48.091457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.091389    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86"} err="failed to get container status \"dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86\": rpc error: code = NotFound desc = could not find container \"dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86\": container with ID starting with dd0b13f6598ffce8de0aadc8433840930fe6c32ea7975ddb5fe2f67adef87e86 not found: ID does not exist"
Nov 02 17:49:48.108820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.108723    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:48.108820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.108753    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcsfd\" (UniqueName: \"kubernetes.io/projected/a6b1b900-6287-41ff-b663-c19037b12bdf-kube-api-access-pcsfd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:48.108820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.108765    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b1b900-6287-41ff-b663-c19037b12bdf-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:48.251081 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.251043    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"]
Nov 02 17:49:48.254948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.254920    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-494cff-77fdb8b6df-q882t"]
Nov 02 17:49:48.689866 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.689835    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"]
Nov 02 17:49:48.690138 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.690068    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" containerID="cri-o://390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b" gracePeriod=30
Nov 02 17:49:48.929162 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:48.929131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:49.384122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:49.384091    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b1b900-6287-41ff-b663-c19037b12bdf" path="/var/lib/kubelet/pods/a6b1b900-6287-41ff-b663-c19037b12bdf/volumes"
Nov 02 17:49:49.736076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:49.735981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:50.524189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:50.524161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:50.754073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:50.754041    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:53.681066 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:53.681032    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:49:53.681468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:53.681103    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:49:53.996354 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:53.996322    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:49:54.099479 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099445    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099494    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099533    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099570    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099729    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099774    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099880    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:54.099988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099913    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.099988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099964    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:54.099988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.099979    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.100196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100013    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy\") pod \"277a2900-6b55-4c0a-a759-30f31a5061e2\" (UID: \"277a2900-6b55-4c0a-a759-30f31a5061e2\") "
Nov 02 17:49:54.100196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100049    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:54.100196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100085    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:54.100332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100269    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:54.100762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100722    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.100762 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100750    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277a2900-6b55-4c0a-a759-30f31a5061e2-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.101104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100767    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.101104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100782    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.101104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.100797    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.101965 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.101933    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "kube-api-access-lxj9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:54.102075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.101994    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:54.102075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.102013    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:54.102517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.102499    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy" pod "277a2900-6b55-4c0a-a759-30f31a5061e2" (UID: "277a2900-6b55-4c0a-a759-30f31a5061e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:54.135681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.129175    4476 generic.go:356] "Generic (PLEG): container finished" podID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerID="390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b" exitCode=0
Nov 02 17:49:54.135681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.129441    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" event={"ID":"277a2900-6b55-4c0a-a759-30f31a5061e2","Type":"ContainerDied","Data":"390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b"}
Nov 02 17:49:54.135681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.129548    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4" event={"ID":"277a2900-6b55-4c0a-a759-30f31a5061e2","Type":"ContainerDied","Data":"70ced4102f4a3f96c497d09e3d1bb3d0e3340ff31cea6d928a5caece2fea2ff4"}
Nov 02 17:49:54.135681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.129594    4476 scope.go:122] "RemoveContainer" containerID="390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b"
Nov 02 17:49:54.135681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.130046    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"
Nov 02 17:49:54.149104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.149077    4476 scope.go:122] "RemoveContainer" containerID="390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b"
Nov 02 17:49:54.149463 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:49:54.149428    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b\": container with ID starting with 390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b not found: ID does not exist" containerID="390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b"
Nov 02 17:49:54.149569 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.149475    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b"} err="failed to get container status \"390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b\": rpc error: code = NotFound desc = could not find container \"390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b\": container with ID starting with 390c25c8f20ff1557e9a9d9cb2f68e66366ed4b0a37f3807c736e73fc04a371b not found: ID does not exist"
Nov 02 17:49:54.202103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.202063    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.202103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.202092    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.202103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.202107    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj9n\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-kube-api-access-lxj9n\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.202325 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.202121    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277a2900-6b55-4c0a-a759-30f31a5061e2-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:49:54.301855 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.301759    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"]
Nov 02 17:49:54.306863 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:54.306836    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-g5hb4"]
Nov 02 17:49:55.112525 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:55.112494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:55.384933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:55.384856    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" path="/var/lib/kubelet/pods/277a2900-6b55-4c0a-a759-30f31a5061e2/volumes"
Nov 02 17:49:57.384552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:57.384515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:59.025191 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:59.025144    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:59.062264 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:49:59.062227    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:49:59.534436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:49:59.534399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:00.645274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:00.645225    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:02.157445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:02.157406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:02.848687 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:02.848537    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:03.244103 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:03.244070    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:06.137158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:06.137111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:07.747578 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:07.747550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:07.977732 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:07.977701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:11.098768 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:11.098738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:11.378682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:11.377714    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:11.755473 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:11.755346    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:16.857286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:16.857256    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:22.183607 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:22.183573    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:24.568662 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:24.568589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:26.284842 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:26.284801    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:27.772799 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:27.772765    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:29.063329 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:50:29.063292    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:50:32.128946 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:32.128852    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"]
Nov 02 17:50:32.499744 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:32.499593    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:32.725063 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:32.725031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:33.389421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:33.389388    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:33.704708 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:33.702114    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:34.625903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:34.625872    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"]
Nov 02 17:50:34.738541 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:34.738511    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"]
Nov 02 17:50:34.746116 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:34.746083    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:35.235967 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:35.235934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:35.484935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:35.484897    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:37.129877 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.129815    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t" podUID="74fa8a46-539d-4688-bdbf-a8dd2597d726" containerName="payload-pre-processing" containerID="cri-o://321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827" gracePeriod=25
Nov 02 17:50:37.393555 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.393498    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:50:37.488021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.487991    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume\") pod \"74fa8a46-539d-4688-bdbf-a8dd2597d726\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") "
Nov 02 17:50:37.488188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.488032    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp\" (UniqueName: \"kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp\") pod \"74fa8a46-539d-4688-bdbf-a8dd2597d726\" (UID: \"74fa8a46-539d-4688-bdbf-a8dd2597d726\") "
Nov 02 17:50:37.488375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.488351    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume" pod "74fa8a46-539d-4688-bdbf-a8dd2597d726" (UID: "74fa8a46-539d-4688-bdbf-a8dd2597d726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:37.490336 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.490314    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp" pod "74fa8a46-539d-4688-bdbf-a8dd2597d726" (UID: "74fa8a46-539d-4688-bdbf-a8dd2597d726"). InnerVolumeSpecName "kube-api-access-jpdzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:37.512263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.512230    4476 generic.go:356] "Generic (PLEG): container finished" podID="74fa8a46-539d-4688-bdbf-a8dd2597d726" containerID="321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827" exitCode=0
Nov 02 17:50:37.512399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.512288    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"
Nov 02 17:50:37.512468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.512403    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t" event={"ID":"74fa8a46-539d-4688-bdbf-a8dd2597d726","Type":"ContainerDied","Data":"321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827"}
Nov 02 17:50:37.512529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.512482    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t" event={"ID":"74fa8a46-539d-4688-bdbf-a8dd2597d726","Type":"ContainerDied","Data":"df0d862d8a8c3f658820808f419c3392e2bec8a1c9baf2b5f7b707e3b3a898c4"}
Nov 02 17:50:37.512529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.512518    4476 scope.go:122] "RemoveContainer" containerID="321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827"
Nov 02 17:50:37.532158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.532118    4476 scope.go:122] "RemoveContainer" containerID="321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827"
Nov 02 17:50:37.532527 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:50:37.532506    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827\": container with ID starting with 321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827 not found: ID does not exist" containerID="321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827"
Nov 02 17:50:37.532605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.532542    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827"} err="failed to get container status \"321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827\": rpc error: code = NotFound desc = could not find container \"321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827\": container with ID starting with 321399d8670cf06abfd557cc486224b1bf93132ec576b65c92b8fa928a1a8827 not found: ID does not exist"
Nov 02 17:50:37.590076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.590036    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74fa8a46-539d-4688-bdbf-a8dd2597d726-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:37.590076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.590067    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdzp\" (UniqueName: \"kubernetes.io/projected/74fa8a46-539d-4688-bdbf-a8dd2597d726-kube-api-access-jpdzp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:37.682760 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.682659    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"]
Nov 02 17:50:37.686252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:37.686229    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-215cbd-59859fdf79-wkx4t"]
Nov 02 17:50:39.225218 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:39.225178    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:39.385299 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:39.385264    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fa8a46-539d-4688-bdbf-a8dd2597d726" path="/var/lib/kubelet/pods/74fa8a46-539d-4688-bdbf-a8dd2597d726/volumes"
Nov 02 17:50:39.626444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:39.626399    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr" podUID="9a7dcd12-61aa-4d4f-986e-95cbd703b15a" containerName="payload-processing" containerID="cri-o://5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c" gracePeriod=25
Nov 02 17:50:39.729498 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:39.729465    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:39.739403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:39.739364    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w" podUID="9a16278d-c430-45c2-9118-7f22ce22b117" containerName="payload-pre-processing" containerID="cri-o://fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a" gracePeriod=25
Nov 02 17:50:39.921591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:39.919434    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:50:40.017095 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.017067    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume\") pod \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") "
Nov 02 17:50:40.017286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.017146    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb\" (UniqueName: \"kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb\") pod \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\" (UID: \"9a7dcd12-61aa-4d4f-986e-95cbd703b15a\") "
Nov 02 17:50:40.017511 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.017482    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume" pod "9a7dcd12-61aa-4d4f-986e-95cbd703b15a" (UID: "9a7dcd12-61aa-4d4f-986e-95cbd703b15a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:40.017699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.017659    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:40.019839 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.019817    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb" pod "9a7dcd12-61aa-4d4f-986e-95cbd703b15a" (UID: "9a7dcd12-61aa-4d4f-986e-95cbd703b15a"). InnerVolumeSpecName "kube-api-access-kksrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:40.040599 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.040571    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:50:40.118363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.118331    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7\" (UniqueName: \"kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7\") pod \"9a16278d-c430-45c2-9118-7f22ce22b117\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") "
Nov 02 17:50:40.118532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.118467    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume\") pod \"9a16278d-c430-45c2-9118-7f22ce22b117\" (UID: \"9a16278d-c430-45c2-9118-7f22ce22b117\") "
Nov 02 17:50:40.118822 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.118806    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kksrb\" (UniqueName: \"kubernetes.io/projected/9a7dcd12-61aa-4d4f-986e-95cbd703b15a-kube-api-access-kksrb\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:40.118914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.118826    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume" pod "9a16278d-c430-45c2-9118-7f22ce22b117" (UID: "9a16278d-c430-45c2-9118-7f22ce22b117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:40.120726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.120701    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7" pod "9a16278d-c430-45c2-9118-7f22ce22b117" (UID: "9a16278d-c430-45c2-9118-7f22ce22b117"). InnerVolumeSpecName "kube-api-access-hstr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:40.219802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.219718    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstr7\" (UniqueName: \"kubernetes.io/projected/9a16278d-c430-45c2-9118-7f22ce22b117-kube-api-access-hstr7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:40.219802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.219751    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a16278d-c430-45c2-9118-7f22ce22b117-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:40.537170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.537078    4476 generic.go:356] "Generic (PLEG): container finished" podID="9a16278d-c430-45c2-9118-7f22ce22b117" containerID="fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a" exitCode=0
Nov 02 17:50:40.537170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.537137    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"
Nov 02 17:50:40.537170 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.537140    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w" event={"ID":"9a16278d-c430-45c2-9118-7f22ce22b117","Type":"ContainerDied","Data":"fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a"}
Nov 02 17:50:40.537746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.537193    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w" event={"ID":"9a16278d-c430-45c2-9118-7f22ce22b117","Type":"ContainerDied","Data":"b92af612016d5700e119dc5ad275db92184449453a0bc1b38aaf9b049023b0e7"}
Nov 02 17:50:40.537746 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.537219    4476 scope.go:122] "RemoveContainer" containerID="fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a"
Nov 02 17:50:40.539115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.539091    4476 generic.go:356] "Generic (PLEG): container finished" podID="9a7dcd12-61aa-4d4f-986e-95cbd703b15a" containerID="5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c" exitCode=0
Nov 02 17:50:40.539233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.539150    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr" event={"ID":"9a7dcd12-61aa-4d4f-986e-95cbd703b15a","Type":"ContainerDied","Data":"5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c"}
Nov 02 17:50:40.539233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.539200    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr" event={"ID":"9a7dcd12-61aa-4d4f-986e-95cbd703b15a","Type":"ContainerDied","Data":"faabd02763411c6115c888fbb57fa6598ec37a2fc6492eb76d9a4bf2e222e77d"}
Nov 02 17:50:40.539233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.539161    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"
Nov 02 17:50:40.550358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.550335    4476 scope.go:122] "RemoveContainer" containerID="fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a"
Nov 02 17:50:40.550676 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:50:40.550657    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a\": container with ID starting with fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a not found: ID does not exist" containerID="fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a"
Nov 02 17:50:40.550757 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.550682    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a"} err="failed to get container status \"fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a\": rpc error: code = NotFound desc = could not find container \"fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a\": container with ID starting with fa41ca5babb3c31e81216bcd0334a9d9b56c716d71d09f9025a212ee8a0d022a not found: ID does not exist"
Nov 02 17:50:40.550757 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.550699    4476 scope.go:122] "RemoveContainer" containerID="5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c"
Nov 02 17:50:40.559912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.559887    4476 scope.go:122] "RemoveContainer" containerID="5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c"
Nov 02 17:50:40.560171 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:50:40.560152    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c\": container with ID starting with 5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c not found: ID does not exist" containerID="5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c"
Nov 02 17:50:40.560256 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.560175    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c"} err="failed to get container status \"5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c\": rpc error: code = NotFound desc = could not find container \"5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c\": container with ID starting with 5534c2e5d4466df71b8e6909edb8b247ecbc942360f1db3db8f7667f11fceb5c not found: ID does not exist"
Nov 02 17:50:40.703348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.703316    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"]
Nov 02 17:50:40.707036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.707013    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-59dccf59cf-zrp2w"]
Nov 02 17:50:40.814364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.814272    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"]
Nov 02 17:50:40.819221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:40.819195    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-6b49776df-l54cr"]
Nov 02 17:50:41.384041 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:41.384007    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a16278d-c430-45c2-9118-7f22ce22b117" path="/var/lib/kubelet/pods/9a16278d-c430-45c2-9118-7f22ce22b117/volumes"
Nov 02 17:50:41.384428 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:41.384414    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7dcd12-61aa-4d4f-986e-95cbd703b15a" path="/var/lib/kubelet/pods/9a7dcd12-61aa-4d4f-986e-95cbd703b15a/volumes"
Nov 02 17:50:44.756156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:44.756115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:49.270235 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:49.270148    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:49.482145 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:49.482101    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:49.867785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:49.867754    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:50.110375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:50.110346    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:50.366885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:50.366800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:54.996686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:54.996582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:55.585300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585269    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"]
Nov 02 17:50:55.585857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585843    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9a7dcd12-61aa-4d4f-986e-95cbd703b15a" containerName="payload-processing"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585858    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7dcd12-61aa-4d4f-986e-95cbd703b15a" containerName="payload-processing"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585869    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a6b1b900-6287-41ff-b663-c19037b12bdf" containerName="maas-api"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585874    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b1b900-6287-41ff-b663-c19037b12bdf" containerName="maas-api"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585893    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9a16278d-c430-45c2-9118-7f22ce22b117" containerName="payload-pre-processing"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585899    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a16278d-c430-45c2-9118-7f22ce22b117" containerName="payload-pre-processing"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585908    4476 cpu_manager.go:414] "containerMap: removing container" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585913    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585919    4476 cpu_manager.go:414] "containerMap: removing container" podUID="74fa8a46-539d-4688-bdbf-a8dd2597d726" containerName="payload-pre-processing"
Nov 02 17:50:55.585925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.585924    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fa8a46-539d-4688-bdbf-a8dd2597d726" containerName="payload-pre-processing"
Nov 02 17:50:55.586241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.586003    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7dcd12-61aa-4d4f-986e-95cbd703b15a" containerName="payload-processing"
Nov 02 17:50:55.586241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.586011    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b1b900-6287-41ff-b663-c19037b12bdf" containerName="maas-api"
Nov 02 17:50:55.586241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.586020    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a16278d-c430-45c2-9118-7f22ce22b117" containerName="payload-pre-processing"
Nov 02 17:50:55.586241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.586027    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="277a2900-6b55-4c0a-a759-30f31a5061e2" containerName="istio-proxy"
Nov 02 17:50:55.586241 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.586032    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fa8a46-539d-4688-bdbf-a8dd2597d726" containerName="payload-pre-processing"
Nov 02 17:50:55.591021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.591001    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.643013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.642988    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8cbedb9f-openshift-default-dockercfg-f6kqf\""
Nov 02 17:50:55.697930 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.697896    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"]
Nov 02 17:50:55.714995 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.714968    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715163 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715006    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ntr\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715163 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715131    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715290 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715208    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715290 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715252    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715378    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715454 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715408    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715543    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.715668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.715603    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817410 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817380    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817439    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817466    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817505    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ntr\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817573    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817690    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.817731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817736    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.818117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817815    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.818117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817845    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.818117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.817890    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.818289 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.818220    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.819166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.818778    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.819166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.818796    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.819166 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.819002    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.821366 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.821136    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.821366 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.821350    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.877003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.876979    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.877845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.877825    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ntr\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:55.904554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:55.904528    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:56.342135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.342107    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"]
Nov 02 17:50:56.342832 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:50:56.342797    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dd4137_bcdc_4743_871b_1b9bd15ea461.slice/crio-8d09834b5985cd1761b8b9225689630851fb2c00c53df0c69b85fc2eafd119fc WatchSource:0}: Error finding container 8d09834b5985cd1761b8b9225689630851fb2c00c53df0c69b85fc2eafd119fc: Status 404 returned error can't find the container with id 8d09834b5985cd1761b8b9225689630851fb2c00c53df0c69b85fc2eafd119fc
Nov 02 17:50:56.345340 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.345309    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:50:56.345425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.345375    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:50:56.345425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.345406    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:50:56.667790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.667723    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" event={"ID":"74dd4137-bcdc-4743-871b-1b9bd15ea461","Type":"ContainerStarted","Data":"73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db"}
Nov 02 17:50:56.667790 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.667793    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" event={"ID":"74dd4137-bcdc-4743-871b-1b9bd15ea461","Type":"ContainerStarted","Data":"8d09834b5985cd1761b8b9225689630851fb2c00c53df0c69b85fc2eafd119fc"}
Nov 02 17:50:56.791919 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.791869    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podStartSLOduration=1.791854266 podStartE2EDuration="1.791854266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:56.787882565 +0000 UTC m=+1687.628488500" watchObservedRunningTime="2026-11-02 17:50:56.791854266 +0000 UTC m=+1687.632460200"
Nov 02 17:50:56.905195 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:56.905165    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:50:57.906811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:57.906774    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 17:50:57.907271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:57.906834    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 17:50:58.905404 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:58.905366    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 17:50:58.905560 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:58.905431    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 17:50:59.082708 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:50:59.082661    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:50:59.168883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.168802    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"]
Nov 02 17:50:59.169037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.169018    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" podUID="8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" containerName="maas-api" containerID="cri-o://7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767" gracePeriod=30
Nov 02 17:50:59.365642 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.365584    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:59.586658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.584444    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:50:59.726671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.725044    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs\") pod \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") "
Nov 02 17:50:59.726671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.725274    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca\") pod \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") "
Nov 02 17:50:59.726671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.725315    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\") pod \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") "
Nov 02 17:50:59.726671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.725375    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k\" (UniqueName: \"kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k\") pod \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\" (UID: \"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8\") "
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.729809    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca" pod "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" (UID: "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.731412    4476 generic.go:356] "Generic (PLEG): container finished" podID="8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" containerID="7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767" exitCode=0
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.731488    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" event={"ID":"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8","Type":"ContainerDied","Data":"7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767"}
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.731538    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr" event={"ID":"8c3f967a-b00f-4c10-9b5f-5d5d0e077de8","Type":"ContainerDied","Data":"81a2d9633bbac1e589188d0ae97ad8032ad80f852179e7ede2a783d1af4c0d86"}
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.731569    4476 scope.go:122] "RemoveContainer" containerID="7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767"
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.731753    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.734497    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k" pod "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" (UID: "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8"). InnerVolumeSpecName "kube-api-access-bt24k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.737180    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs" pod "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" (UID: "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:50:59.747659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.746120    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls" pod "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" (UID: "8c3f967a-b00f-4c10-9b5f-5d5d0e077de8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:50:59.787791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.787765    4476 scope.go:122] "RemoveContainer" containerID="7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767"
Nov 02 17:50:59.788313 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:50:59.788256    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767\": container with ID starting with 7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767 not found: ID does not exist" containerID="7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767"
Nov 02 17:50:59.788313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.788291    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767"} err="failed to get container status \"7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767\": rpc error: code = NotFound desc = could not find container \"7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767\": container with ID starting with 7ce9e58dc1abeeb5a0936694af6498577f15ece091ac65ec1bcb4b972d47e767 not found: ID does not exist"
Nov 02 17:50:59.827362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.827295    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt24k\" (UniqueName: \"kubernetes.io/projected/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-kube-api-access-bt24k\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:59.827362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.827328    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:59.827362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.827340    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:59.827362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.827349    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:50:59.905488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.905459    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 17:50:59.905651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:50:59.905513    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 17:51:00.202716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:00.202681    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"]
Nov 02 17:51:00.244311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:00.240120    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-215cbd-745df5cbc5-czlkr"]
Nov 02 17:51:00.905858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:00.905815    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 17:51:00.906017 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:00.905878    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 17:51:01.199598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:01.199512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:01.385250 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:01.385218    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" path="/var/lib/kubelet/pods/8c3f967a-b00f-4c10-9b5f-5d5d0e077de8/volumes"
Nov 02 17:51:01.905266 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:01.905220    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 17:51:01.905493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:01.905293    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 17:51:02.909895 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:02.909867    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:51:02.910347 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:02.910173    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:51:02.910928 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:02.910909    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:51:05.089768 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:05.089736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:05.465457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:05.465429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:06.118655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:06.118602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:06.403164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:06.403084    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:07.477488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:07.477452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:08.151405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:08.151348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:09.055119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:09.055079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:10.814357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.814320    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"]
Nov 02 17:51:10.815259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.815235    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" containerName="maas-api"
Nov 02 17:51:10.815373 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.815259    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" containerName="maas-api"
Nov 02 17:51:10.815433 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.815423    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3f967a-b00f-4c10-9b5f-5d5d0e077de8" containerName="maas-api"
Nov 02 17:51:10.819513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.819462    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.850528 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.850500    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"]
Nov 02 17:51:10.854760 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.854736    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:10.896330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896291    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.896510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896442    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.896510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896488    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.896642 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896544    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.896709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896642    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.896880 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896840    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.897006 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.896984    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6626\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.897150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.897034    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.897150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.897060    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:10.917093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.917067    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"]
Nov 02 17:51:10.967383 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:10.967348    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"]
Nov 02 17:51:11.000591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000549    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.000591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000593    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.000860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.000860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000739    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.000860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000765    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.000860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000806    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.001087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000884    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.001087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.000978    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.001087 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.001079    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.001642 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.001604    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.001797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.001775    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.001928    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.001950    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.001980    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6626\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002066    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002108    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.002370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002295    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.002583 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002549    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.002724 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002616    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.002811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002789    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v4px\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.002937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.002918    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.003328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.003293    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.004316 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.004285    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.005591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.005567    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.072969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.070759    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6626\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.081917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.081885    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.103894 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.103860    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.103953    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.103978    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104049    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104071    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104087    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104114    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v4px\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104151    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104177    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104583 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104564    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.104922 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.104892    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.105493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.105169    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.105493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.105445    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.105730 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.105540    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.107845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.107782    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.108622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.108595    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.135975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.135950    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:11.182866 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.182826    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.216294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.216258    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v4px\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.468715 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.468681    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:11.583268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.583233    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"]
Nov 02 17:51:11.584002 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:51:11.583975    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275e7402_8cf3_488c_95ed_e0da02d3dafe.slice/crio-396e43743f48bb5e0b8a1985b75aacfd16ef5eca8a72833b94ae1cdec59ecc3e WatchSource:0}: Error finding container 396e43743f48bb5e0b8a1985b75aacfd16ef5eca8a72833b94ae1cdec59ecc3e: Status 404 returned error can't find the container with id 396e43743f48bb5e0b8a1985b75aacfd16ef5eca8a72833b94ae1cdec59ecc3e
Nov 02 17:51:11.586611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.586577    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:11.586728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.586702    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:11.586793 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.586763    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:11.851071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.850976    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" event={"ID":"275e7402-8cf3-488c-95ed-e0da02d3dafe","Type":"ContainerStarted","Data":"40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1"}
Nov 02 17:51:11.851071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.851042    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" event={"ID":"275e7402-8cf3-488c-95ed-e0da02d3dafe","Type":"ContainerStarted","Data":"396e43743f48bb5e0b8a1985b75aacfd16ef5eca8a72833b94ae1cdec59ecc3e"}
Nov 02 17:51:11.869420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.869390    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"]
Nov 02 17:51:11.872969 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:51:11.872944    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3910b518_983d_4b42_98c4_f3af2188d031.slice/crio-5ba79d3e40954b0f571df11280c7eeb8b3d357a237be1b23d73540dfaa13b699 WatchSource:0}: Error finding container 5ba79d3e40954b0f571df11280c7eeb8b3d357a237be1b23d73540dfaa13b699: Status 404 returned error can't find the container with id 5ba79d3e40954b0f571df11280c7eeb8b3d357a237be1b23d73540dfaa13b699
Nov 02 17:51:11.876133 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.876096    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:11.876214 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.876177    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:11.876265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:11.876220    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:12.136801 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:12.136767    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:12.141395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:12.141167    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:12.141395 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:12.141221    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:12.198818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:12.198788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:12.893963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:12.893802    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" event={"ID":"3910b518-983d-4b42-98c4-f3af2188d031","Type":"ContainerStarted","Data":"5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771"}
Nov 02 17:51:12.893963 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:12.893890    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" event={"ID":"3910b518-983d-4b42-98c4-f3af2188d031","Type":"ContainerStarted","Data":"5ba79d3e40954b0f571df11280c7eeb8b3d357a237be1b23d73540dfaa13b699"}
Nov 02 17:51:13.131772 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.131718    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podStartSLOduration=3.131701955 podStartE2EDuration="3.131701955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:11.989920151 +0000 UTC m=+1702.830526090" watchObservedRunningTime="2026-11-02 17:51:13.131701955 +0000 UTC m=+1703.972307899"
Nov 02 17:51:13.134044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.134011    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podStartSLOduration=3.134001413 podStartE2EDuration="3.134001413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:13.08125655 +0000 UTC m=+1703.921862485" watchObservedRunningTime="2026-11-02 17:51:13.134001413 +0000 UTC m=+1703.974607355"
Nov 02 17:51:13.136981 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.136959    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:13.137031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.137013    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:13.486656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.475001    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:13.486656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.480747    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 17:51:13.486656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:13.480802    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 17:51:14.137327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:14.137289    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:14.137786 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:14.137377    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:14.476716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:14.476061    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 17:51:14.476716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:14.476126    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 17:51:15.136597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:15.136551    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:15.136792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:15.136642    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:15.472708 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:15.472038    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 17:51:15.472708 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:15.472096    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 17:51:16.136684 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:16.136649    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:16.136861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:16.136705    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:16.476713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:16.475973    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 17:51:16.476713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:16.476041    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 17:51:16.677067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:16.677030    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"]
Nov 02 17:51:16.787711 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:16.783713    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"]
Nov 02 17:51:17.177655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:17.176774    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:17.177655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:17.176838    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:17.198484 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:17.198406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:17.471704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:17.471547    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 17:51:17.471704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:17.471622    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 17:51:17.503922 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:17.499479    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:18.145659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:18.138049    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 02 17:51:18.145659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:18.138115    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 02 17:51:18.336608 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:18.336580    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:18.470134 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:18.470031    4476 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 02 17:51:18.470134 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:18.470080    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 02 17:51:19.142218 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:19.142187    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:19.142724 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:19.142662    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:19.143420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:19.143401    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:51:19.489003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:19.488923    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:19.489842 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:19.489809    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:19.490798 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:19.490772    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:51:21.677758 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.677719    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm" podUID="fc811840-7d78-4c9d-b29f-927c1c411944" containerName="payload-processing" containerID="cri-o://89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b" gracePeriod=25
Nov 02 17:51:21.714338 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:51:21.714307    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc811840_7d78_4c9d_b29f_927c1c411944.slice/crio-conmon-89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc811840_7d78_4c9d_b29f_927c1c411944.slice/crio-89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 17:51:21.714493 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:51:21.714463    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc811840_7d78_4c9d_b29f_927c1c411944.slice/crio-89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc811840_7d78_4c9d_b29f_927c1c411944.slice/crio-conmon-89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 17:51:21.785448 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.785379    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6" podUID="994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" containerName="payload-pre-processing" containerID="cri-o://d857f1c15b05427ae83c460c6c23b676ee141f4dd3db8f8c4c1ae8e3c08fe0bf" gracePeriod=25
Nov 02 17:51:21.974209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.971982    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:51:21.974209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.973928    4476 generic.go:356] "Generic (PLEG): container finished" podID="fc811840-7d78-4c9d-b29f-927c1c411944" containerID="89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b" exitCode=0
Nov 02 17:51:21.974209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.973999    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm" event={"ID":"fc811840-7d78-4c9d-b29f-927c1c411944","Type":"ContainerDied","Data":"89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b"}
Nov 02 17:51:21.974209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.974054    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm" event={"ID":"fc811840-7d78-4c9d-b29f-927c1c411944","Type":"ContainerDied","Data":"64fe906f505f94b4264d60e6e0e665f02f3b86e663b3584e47ba64ecb80a006e"}
Nov 02 17:51:21.974209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.974077    4476 scope.go:122] "RemoveContainer" containerID="89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b"
Nov 02 17:51:21.979495 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.977420    4476 generic.go:356] "Generic (PLEG): container finished" podID="994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" containerID="d857f1c15b05427ae83c460c6c23b676ee141f4dd3db8f8c4c1ae8e3c08fe0bf" exitCode=0
Nov 02 17:51:21.979495 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.977574    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6" event={"ID":"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0","Type":"ContainerDied","Data":"d857f1c15b05427ae83c460c6c23b676ee141f4dd3db8f8c4c1ae8e3c08fe0bf"}
Nov 02 17:51:21.994412 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.994383    4476 scope.go:122] "RemoveContainer" containerID="89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b"
Nov 02 17:51:21.999867 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:51:21.999777    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b\": container with ID starting with 89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b not found: ID does not exist" containerID="89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b"
Nov 02 17:51:21.999867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:21.999822    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b"} err="failed to get container status \"89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b\": rpc error: code = NotFound desc = could not find container \"89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b\": container with ID starting with 89616876a0bc8f1c77ba28214e1fbd45d2d53c00f6890997084352b6ae126b8b not found: ID does not exist"
Nov 02 17:51:22.069332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.069307    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:51:22.084724 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.084698    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume\") pod \"fc811840-7d78-4c9d-b29f-927c1c411944\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") "
Nov 02 17:51:22.084831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.084757    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r\" (UniqueName: \"kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r\") pod \"fc811840-7d78-4c9d-b29f-927c1c411944\" (UID: \"fc811840-7d78-4c9d-b29f-927c1c411944\") "
Nov 02 17:51:22.085080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.085057    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume" pod "fc811840-7d78-4c9d-b29f-927c1c411944" (UID: "fc811840-7d78-4c9d-b29f-927c1c411944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:22.085477 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.085461    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc811840-7d78-4c9d-b29f-927c1c411944-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:22.087018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.086999    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r" pod "fc811840-7d78-4c9d-b29f-927c1c411944" (UID: "fc811840-7d78-4c9d-b29f-927c1c411944"). InnerVolumeSpecName "kube-api-access-qml9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:22.186264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.186233    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume\") pod \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") "
Nov 02 17:51:22.186264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.186269    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948\" (UniqueName: \"kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948\") pod \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\" (UID: \"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0\") "
Nov 02 17:51:22.186705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.186682    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qml9r\" (UniqueName: \"kubernetes.io/projected/fc811840-7d78-4c9d-b29f-927c1c411944-kube-api-access-qml9r\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:22.186705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.186676    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume" pod "994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" (UID: "994ecfe6-ca82-4dae-8ebf-e8c4675c9af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:22.188541 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.188520    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948" pod "994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" (UID: "994ecfe6-ca82-4dae-8ebf-e8c4675c9af0"). InnerVolumeSpecName "kube-api-access-qn948". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:22.288424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.288334    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:22.288424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:22.288368    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn948\" (UniqueName: \"kubernetes.io/projected/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0-kube-api-access-qn948\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:51:23.049115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.049020    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"
Nov 02 17:51:23.066277 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.066236    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6" event={"ID":"994ecfe6-ca82-4dae-8ebf-e8c4675c9af0","Type":"ContainerDied","Data":"d10f8efaf4e415cf684c02efcba861ee850a09b53449bffbc5f4349998d0e457"}
Nov 02 17:51:23.066540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.066525    4476 scope.go:122] "RemoveContainer" containerID="d857f1c15b05427ae83c460c6c23b676ee141f4dd3db8f8c4c1ae8e3c08fe0bf"
Nov 02 17:51:23.066768 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.066755    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"
Nov 02 17:51:23.135754 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.135705    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:23.296653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.293801    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"]
Nov 02 17:51:23.341411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.339665    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f0f3e1-576ff84b55-p4wcm"]
Nov 02 17:51:23.443654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.439907    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc811840-7d78-4c9d-b29f-927c1c411944" path="/var/lib/kubelet/pods/fc811840-7d78-4c9d-b29f-927c1c411944/volumes"
Nov 02 17:51:23.483649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.479414    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"]
Nov 02 17:51:23.483649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.479479    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f0f3e1-754c46b47f-c9tb6"]
Nov 02 17:51:23.933092 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:23.933060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:25.371707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:25.371677    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:25.387204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:25.387170    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" path="/var/lib/kubelet/pods/994ecfe6-ca82-4dae-8ebf-e8c4675c9af0/volumes"
Nov 02 17:51:29.060419 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:51:29.060378    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:51:30.255301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:30.255270    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:30.517277 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:30.517183    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:31.004219 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:31.004181    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:32.215357 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:32.215322    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:32.850853 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:32.850816    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:33.152295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:33.152250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:34.002590 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:34.002539    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:34.831416 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:34.831367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:35.061205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:35.061154    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:35.624389 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:35.624358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:38.191642 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:38.191589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:38.497423 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:38.497300    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:39.923121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:39.923085    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:40.166121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:40.166094    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:40.401213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:40.401185    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:43.344396 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.344358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:43.578830 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.578775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:43.713358 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.713324    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"]
Nov 02 17:51:43.714374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.714162    4476 cpu_manager.go:414] "containerMap: removing container" podUID="994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" containerName="payload-pre-processing"
Nov 02 17:51:43.714374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.714178    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" containerName="payload-pre-processing"
Nov 02 17:51:43.714374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.714189    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fc811840-7d78-4c9d-b29f-927c1c411944" containerName="payload-processing"
Nov 02 17:51:43.714374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.714194    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc811840-7d78-4c9d-b29f-927c1c411944" containerName="payload-processing"
Nov 02 17:51:43.714374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.714296    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="994ecfe6-ca82-4dae-8ebf-e8c4675c9af0" containerName="payload-pre-processing"
Nov 02 17:51:43.714374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.714303    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc811840-7d78-4c9d-b29f-927c1c411944" containerName="payload-processing"
Nov 02 17:51:43.717882 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.717852    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.774048 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.774013    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.774265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.774069    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrdz\" (UniqueName: \"kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz\") pod \"payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.783031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.782994    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"]
Nov 02 17:51:43.822833 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.822791    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-700abe-dockercfg-6f5jn\""
Nov 02 17:51:43.823059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.823048    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-700abe\""
Nov 02 17:51:43.879417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.879356    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.879778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.879753    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrdz\" (UniqueName: \"kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz\") pod \"payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.892878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.883348    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"]
Nov 02 17:51:43.892878 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.888428    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:43.899368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.899340    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.923905 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.923154    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"]
Nov 02 17:51:43.958899 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.950751    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrdz\" (UniqueName: \"kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz\") pod \"payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:43.985771 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.985565    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:43.985947 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:43.985789    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktd5\" (UniqueName: \"kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5\") pod \"payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:44.033052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.032991    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:44.087222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.087180    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:44.087409 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.087368    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktd5\" (UniqueName: \"kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5\") pod \"payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:44.088345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.088315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:44.145748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.145709    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktd5\" (UniqueName: \"kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5\") pod \"payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:44.250523 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.250438    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:44.275607 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.275573    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:44.432675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.432644    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"]
Nov 02 17:51:44.433251 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:51:44.433229    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fea00c_064b_48ed_a8aa_612704e68898.slice/crio-553b5d8297481b0f47b68378f44743dc494a897b046ee75ae0889115e3c985bb WatchSource:0}: Error finding container 553b5d8297481b0f47b68378f44743dc494a897b046ee75ae0889115e3c985bb: Status 404 returned error can't find the container with id 553b5d8297481b0f47b68378f44743dc494a897b046ee75ae0889115e3c985bb
Nov 02 17:51:44.663332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:44.663304    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"]
Nov 02 17:51:45.234975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.234945    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc" event={"ID":"14fea00c-064b-48ed-a8aa-612704e68898","Type":"ContainerStarted","Data":"b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818"}
Nov 02 17:51:45.235188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.234997    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc" event={"ID":"14fea00c-064b-48ed-a8aa-612704e68898","Type":"ContainerStarted","Data":"553b5d8297481b0f47b68378f44743dc494a897b046ee75ae0889115e3c985bb"}
Nov 02 17:51:45.235188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.235022    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:45.236837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.236812    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls" event={"ID":"962e1b3b-adb7-464f-bac1-cfd66c443195","Type":"ContainerStarted","Data":"f9d0aec37739d80c3044e76f0a717e7b646273386f8bd3927f682b4c16a87144"}
Nov 02 17:51:45.236966 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.236855    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls" event={"ID":"962e1b3b-adb7-464f-bac1-cfd66c443195","Type":"ContainerStarted","Data":"08515f74b9bb2ecbb634692b83a84dd1c10ad90594573d3fe94fad86a177581c"}
Nov 02 17:51:45.236966 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.236903    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:45.359607 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.359559    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc" podStartSLOduration=2.3595462449999998 podStartE2EDuration="2.359546245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:45.356964185 +0000 UTC m=+1736.197570132" watchObservedRunningTime="2026-11-02 17:51:45.359546245 +0000 UTC m=+1736.200152200"
Nov 02 17:51:45.491850 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:45.491730    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls" podStartSLOduration=2.4917145339999998 podStartE2EDuration="2.491714534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:45.488070295 +0000 UTC m=+1736.328676229" watchObservedRunningTime="2026-11-02 17:51:45.491714534 +0000 UTC m=+1736.332320465"
Nov 02 17:51:48.744231 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.744196    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"]
Nov 02 17:51:48.748432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.748413    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.849681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.849623    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6f23bd-openshift-default-dockercfg-9wmkr\""
Nov 02 17:51:48.853899 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.853866    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854079 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.853924    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854110    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854169    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854251    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854404 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854283    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854470 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854410    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhh9\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854470 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854452    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.854571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.854529    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.856914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.856890    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"]
Nov 02 17:51:48.955979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.955923    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956036    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956083    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956276 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956212    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956378 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956359    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956454 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956417    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956460    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956535    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956546    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956690 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956561    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956690 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956610    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhh9\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956844    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.956931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.956914    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.957138 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.957107    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.958871 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.958844    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:48.959263 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:48.959242    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:49.029300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.029213    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhh9\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:49.029300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.029286    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:49.064411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.064378    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:49.317896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.317805    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:49.503862 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.503833    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"]
Nov 02 17:51:49.504828 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:51:49.504804    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecda5590_9500_4f46_a220_894b04a9bc47.slice/crio-c4dae8aa1918c373d63ecdfa23f89f5a4f3af56c9e2940567eae9ae87e2c2eea WatchSource:0}: Error finding container c4dae8aa1918c373d63ecdfa23f89f5a4f3af56c9e2940567eae9ae87e2c2eea: Status 404 returned error can't find the container with id c4dae8aa1918c373d63ecdfa23f89f5a4f3af56c9e2940567eae9ae87e2c2eea
Nov 02 17:51:49.508057 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.508021    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:49.508138 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.508102    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:49.508172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.508142    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:51:49.605251 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:49.605210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:50.276860 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:50.276822    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" event={"ID":"ecda5590-9500-4f46-a220-894b04a9bc47","Type":"ContainerStarted","Data":"baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af"}
Nov 02 17:51:50.277298 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:50.276924    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" event={"ID":"ecda5590-9500-4f46-a220-894b04a9bc47","Type":"ContainerStarted","Data":"c4dae8aa1918c373d63ecdfa23f89f5a4f3af56c9e2940567eae9ae87e2c2eea"}
Nov 02 17:51:50.498279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:50.498221    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podStartSLOduration=2.498206858 podStartE2EDuration="2.498206858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:50.448123403 +0000 UTC m=+1741.288729351" watchObservedRunningTime="2026-11-02 17:51:50.498206858 +0000 UTC m=+1741.338812793"
Nov 02 17:51:50.971559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:50.971525    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:51.066770 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:51.064992    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:51.069257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:51.069230    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 17:51:51.069389 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:51.069281    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 17:51:51.169672 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:51.169606    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:52.069471 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:52.069433    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 17:51:52.070007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:52.069488    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 17:51:53.066653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:53.065960    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 17:51:53.066653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:53.066018    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 17:51:54.067651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:54.067004    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 17:51:54.067651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:54.067070    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 17:51:54.323057 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:54.322965    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:55.065652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:55.065019    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 02 17:51:55.065652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:55.065093    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 02 17:51:56.070568 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:56.070525    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:56.071070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:56.070837    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:56.071698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:56.071675    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:51:56.248077 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:56.248044    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:51:56.248487 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:56.248466    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:51:58.230068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:58.230038    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:58.976782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:58.976745    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:59.080992 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:51:59.080951    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:51:59.305500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:59.305414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:59.513047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:51:59.513011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:00.218068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:00.218033    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:01.665941 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:01.665906    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:03.469168 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:03.469138    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:03.702696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:03.702537    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:04.885120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:04.885085    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:06.754829 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:06.754792    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:06.944305 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:06.944269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:07.076778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.076148    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"]
Nov 02 17:52:07.083487 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.083201    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.136397 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.136363    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"]
Nov 02 17:52:07.180338 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.180308    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f10385\""
Nov 02 17:52:07.194709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.194675    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmqf\" (UniqueName: \"kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf\") pod \"payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.194844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.194802    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.291075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.291037    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"]
Nov 02 17:52:07.316857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.316816    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmqf\" (UniqueName: \"kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf\") pod \"payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.317265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.317242    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.318324 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.318298    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.329421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.329347    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.340598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.340556    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"]
Nov 02 17:52:07.390480 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.390453    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f10385-dockercfg-pf2xb\""
Nov 02 17:52:07.406104 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.406071    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmqf\" (UniqueName: \"kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf\") pod \"payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.409486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.409450    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:07.418936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.418912    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.419215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.419112    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6m8h\" (UniqueName: \"kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h\") pod \"payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.520942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.520582    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6m8h\" (UniqueName: \"kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h\") pod \"payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.520942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.520792    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.522836 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.522806    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.587731 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.585969    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6m8h\" (UniqueName: \"kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h\") pod \"payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.717805 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.717774    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:07.805201 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:07.799375    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"]
Nov 02 17:52:07.805201 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:52:07.804139    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161cf413_1264_451a_8085_0226d0ef3ca0.slice/crio-956f2ee9288f3db4f1e55c5f9e03e4ee27b0a78c59c0a8e58418dcd4eb397145 WatchSource:0}: Error finding container 956f2ee9288f3db4f1e55c5f9e03e4ee27b0a78c59c0a8e58418dcd4eb397145: Status 404 returned error can't find the container with id 956f2ee9288f3db4f1e55c5f9e03e4ee27b0a78c59c0a8e58418dcd4eb397145
Nov 02 17:52:08.102598 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.102566    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"]
Nov 02 17:52:08.102803 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:52:08.102778    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92aac989_f02c_449e_aa23_ecef70245d73.slice/crio-ee365254c6d364786cf31d12953e316d8430a27c20463f7427c288e3d865e26a WatchSource:0}: Error finding container ee365254c6d364786cf31d12953e316d8430a27c20463f7427c288e3d865e26a: Status 404 returned error can't find the container with id ee365254c6d364786cf31d12953e316d8430a27c20463f7427c288e3d865e26a
Nov 02 17:52:08.323911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.323875    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:08.422342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.422312    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w" event={"ID":"92aac989-f02c-449e-aa23-ecef70245d73","Type":"ContainerStarted","Data":"5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a"}
Nov 02 17:52:08.422519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.422360    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w" event={"ID":"92aac989-f02c-449e-aa23-ecef70245d73","Type":"ContainerStarted","Data":"ee365254c6d364786cf31d12953e316d8430a27c20463f7427c288e3d865e26a"}
Nov 02 17:52:08.422519 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.422462    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:08.424057 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.424033    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g" event={"ID":"161cf413-1264-451a-8085-0226d0ef3ca0","Type":"ContainerStarted","Data":"14cec23926e08f9802e1340ee802cfc7f69a41baf8ae89463649c62d0a3900f0"}
Nov 02 17:52:08.424172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.424073    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g" event={"ID":"161cf413-1264-451a-8085-0226d0ef3ca0","Type":"ContainerStarted","Data":"956f2ee9288f3db4f1e55c5f9e03e4ee27b0a78c59c0a8e58418dcd4eb397145"}
Nov 02 17:52:08.424172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.424093    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:08.539053 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.538993    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w" podStartSLOduration=1.5389752570000002 podStartE2EDuration="1.538975257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:08.536247693 +0000 UTC m=+1759.376853644" watchObservedRunningTime="2026-11-02 17:52:08.538975257 +0000 UTC m=+1759.379581192"
Nov 02 17:52:08.661460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:08.661267    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g" podStartSLOduration=1.661245542 podStartE2EDuration="1.661245542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:08.658187969 +0000 UTC m=+1759.498793916" watchObservedRunningTime="2026-11-02 17:52:08.661245542 +0000 UTC m=+1759.501851477"
Nov 02 17:52:09.886415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:09.886384    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:11.969097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:11.969060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:12.174700 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:12.167458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:14.753875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:14.753836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:15.504679 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:15.504592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:15.771675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:15.771485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:16.392852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:16.392818    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:16.898492 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:16.898455    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"]
Nov 02 17:52:17.250536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:17.250438    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:17.450445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:17.450414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:18.981221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:18.981188    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"]
Nov 02 17:52:18.987569 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:18.987546    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.093515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.093481    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"]
Nov 02 17:52:19.121847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.121807    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.121851    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.121972    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.121997    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.122025    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.122180    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.122229    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122321 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.122257    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27vc\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.122517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.122493    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.223597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.223561    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.223800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.223619    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.223800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.223687    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.223800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.223741    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.223800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.223766    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224022 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.223813    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224071    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224185 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224156    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224255    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224293    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224336    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27vc\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224417    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224514    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.224716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.224691    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.227167 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.227142    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.227464 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.227442    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.282235 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.282157    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27vc\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.282399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.282232    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.302015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.301984    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:19.433510 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.433482    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:52:19.433704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.433550    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:52:19.743386 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.743352    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"]
Nov 02 17:52:19.744854 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:52:19.744819    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7069351_3bef_481e_a809_acc2f2b27c83.slice/crio-5e168cf0bdb0923d1484a675dace1dc71867708cc490f0a4f863c975e075a79b WatchSource:0}: Error finding container 5e168cf0bdb0923d1484a675dace1dc71867708cc490f0a4f863c975e075a79b: Status 404 returned error can't find the container with id 5e168cf0bdb0923d1484a675dace1dc71867708cc490f0a4f863c975e075a79b
Nov 02 17:52:19.747913 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.747881    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:52:19.748013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.747971    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:52:19.748086 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:19.748019    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:52:20.102271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:20.102179    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:20.596122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:20.596080    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" event={"ID":"e7069351-3bef-481e-a809-acc2f2b27c83","Type":"ContainerStarted","Data":"117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104"}
Nov 02 17:52:20.596314 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:20.596144    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" event={"ID":"e7069351-3bef-481e-a809-acc2f2b27c83","Type":"ContainerStarted","Data":"5e168cf0bdb0923d1484a675dace1dc71867708cc490f0a4f863c975e075a79b"}
Nov 02 17:52:20.725557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:20.725499    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podStartSLOduration=2.7254840529999997 podStartE2EDuration="2.725484053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:20.722199284 +0000 UTC m=+1771.562805221" watchObservedRunningTime="2026-11-02 17:52:20.725484053 +0000 UTC m=+1771.566089986"
Nov 02 17:52:21.032414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.032372    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:21.234100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.234060    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:21.302858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.302777    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:21.304782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.304755    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 17:52:21.304913 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.304805    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 17:52:21.898929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.898889    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p" podUID="b16c95fe-e82f-480e-a274-ebf438f12ab0" containerName="payload-pre-processing" containerID="cri-o://da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7" gracePeriod=25
Nov 02 17:52:21.935931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:21.935893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:22.125959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.125921    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:22.332407 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.331566    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 17:52:22.332407 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.331674    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 17:52:22.338034 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.337999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:22.422647 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.420007    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:52:22.524794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.524151    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume\") pod \"b16c95fe-e82f-480e-a274-ebf438f12ab0\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") "
Nov 02 17:52:22.524794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.524229    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc\" (UniqueName: \"kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc\") pod \"b16c95fe-e82f-480e-a274-ebf438f12ab0\" (UID: \"b16c95fe-e82f-480e-a274-ebf438f12ab0\") "
Nov 02 17:52:22.525036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.524800    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume" pod "b16c95fe-e82f-480e-a274-ebf438f12ab0" (UID: "b16c95fe-e82f-480e-a274-ebf438f12ab0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:22.527604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.527577    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc" pod "b16c95fe-e82f-480e-a274-ebf438f12ab0" (UID: "b16c95fe-e82f-480e-a274-ebf438f12ab0"). InnerVolumeSpecName "kube-api-access-pzhgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:22.615249 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.612996    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:22.630792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.626940    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c95fe-e82f-480e-a274-ebf438f12ab0-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.630792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.626973    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhgc\" (UniqueName: \"kubernetes.io/projected/b16c95fe-e82f-480e-a274-ebf438f12ab0-kube-api-access-pzhgc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.632507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.632283    4476 generic.go:356] "Generic (PLEG): container finished" podID="b16c95fe-e82f-480e-a274-ebf438f12ab0" containerID="da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7" exitCode=0
Nov 02 17:52:22.632507 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.632333    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p" event={"ID":"b16c95fe-e82f-480e-a274-ebf438f12ab0","Type":"ContainerDied","Data":"da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7"}
Nov 02 17:52:22.632742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.632528    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p" event={"ID":"b16c95fe-e82f-480e-a274-ebf438f12ab0","Type":"ContainerDied","Data":"5d4d6e00145b84314aa9d00e2d2d63641e3ce7f30eb729c22844ea4462be5ae3"}
Nov 02 17:52:22.632742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.632569    4476 scope.go:122] "RemoveContainer" containerID="da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7"
Nov 02 17:52:22.632742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.632342    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"
Nov 02 17:52:22.697522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.697499    4476 scope.go:122] "RemoveContainer" containerID="da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7"
Nov 02 17:52:22.697945 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:52:22.697920    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7\": container with ID starting with da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7 not found: ID does not exist" containerID="da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7"
Nov 02 17:52:22.698023 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.697958    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7"} err="failed to get container status \"da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7\": rpc error: code = NotFound desc = could not find container \"da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7\": container with ID starting with da0fedfc83e3ee52b656a8c815f687e8161ad80c54c9edfbd3430a86485d85f7 not found: ID does not exist"
Nov 02 17:52:22.879713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.879007    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:22.879713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.879175    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"]
Nov 02 17:52:22.907655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:22.897770    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8cbedb9f-589cd7db79-rv95p"]
Nov 02 17:52:23.218391 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:23.218286    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:23.307028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:23.306542    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 17:52:23.307028 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:23.306604    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 17:52:23.397574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:23.397539    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16c95fe-e82f-480e-a274-ebf438f12ab0" path="/var/lib/kubelet/pods/b16c95fe-e82f-480e-a274-ebf438f12ab0/volumes"
Nov 02 17:52:24.302658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:24.302355    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 17:52:24.302658 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:24.302417    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 17:52:24.529652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:24.527114    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:25.014440 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.014403    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"]
Nov 02 17:52:25.014977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.014657    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy" containerID="cri-o://5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771" gracePeriod=30
Nov 02 17:52:25.023128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.023083    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"]
Nov 02 17:52:25.023537 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.023493    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy" containerID="cri-o://40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1" gracePeriod=30
Nov 02 17:52:25.072142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.072110    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:25.075845 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.075815    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"]
Nov 02 17:52:25.076037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.076016    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy" containerID="cri-o://73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db" gracePeriod=30
Nov 02 17:52:25.302551 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.302463    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 17:52:25.302853 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.302826    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 17:52:25.851653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:25.849192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:26.450541 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:26.450510    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:26.452978 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:26.452951    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:26.455659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:26.455587    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:26.465814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:26.465787    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:52:26.663759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:26.663705    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:28.126540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:28.126483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:28.359291 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:28.359254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:28.586471 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:28.586367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:28.898120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:28.898085    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:29.060366 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:52:29.060326    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:52:30.380540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.380197    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:52:30.407755 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.407557    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:52:30.434644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.434612    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:52:30.447566 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447543    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.447751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447589    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.447751 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447619    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.447962 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447859    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.447962 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447918    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447963    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.448073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.447985    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:30.448073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448032    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.448282 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448181    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448282 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448222    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.448282 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448255    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.448434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448288    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.448434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448380    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:30.448434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448384    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.448592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448532    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448548    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.448720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448613    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448720 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448687    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448757    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448804    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448837    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket\") pod \"275e7402-8cf3-488c-95ed-e0da02d3dafe\" (UID: \"275e7402-8cf3-488c-95ed-e0da02d3dafe\") "
Nov 02 17:52:30.448937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448900    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs\") pod \"3910b518-983d-4b42-98c4-f3af2188d031\" (UID: \"3910b518-983d-4b42-98c4-f3af2188d031\") "
Nov 02 17:52:30.449052 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.448971    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.449443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449400    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.449671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449648    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.449753 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449722    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.449753 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449744    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.449870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449761    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.449870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449775    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b518-983d-4b42-98c4-f3af2188d031-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.449870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449790    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275e7402-8cf3-488c-95ed-e0da02d3dafe-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.449870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.449803    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.450073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.450003    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.450221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.450200    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.450310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.450283    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.450515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.450494    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.450733 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.450706    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "kube-api-access-m6626". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.451029 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.451005    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:30.451188 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.451164    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "kube-api-access-8v4px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.451380 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.451358    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:30.452313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.452293    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token" pod "3910b518-983d-4b42-98c4-f3af2188d031" (UID: "3910b518-983d-4b42-98c4-f3af2188d031"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.452378 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.452357    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.452445 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.452430    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy" pod "275e7402-8cf3-488c-95ed-e0da02d3dafe" (UID: "275e7402-8cf3-488c-95ed-e0da02d3dafe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.559273 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559236    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559321    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559378    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559411    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559489    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559554    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559656    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559797    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.559959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559794    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.559959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.559885    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data\") pod \"74dd4137-bcdc-4743-871b-1b9bd15ea461\" (UID: \"74dd4137-bcdc-4743-871b-1b9bd15ea461\") "
Nov 02 17:52:30.560784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560688    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.560784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560766    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560796    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b518-983d-4b42-98c4-f3af2188d031-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560815    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b518-983d-4b42-98c4-f3af2188d031-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560832    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6626\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-kube-api-access-m6626\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560848    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v4px\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-kube-api-access-8v4px\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560864    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b518-983d-4b42-98c4-f3af2188d031-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560883    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560984    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.560999    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.561015    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.561029    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.561057    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.561045    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.561100    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275e7402-8cf3-488c-95ed-e0da02d3dafe-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.561426 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.561348    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.562345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.562309    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.566687 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.564907    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:30.568046 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.568020    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:30.568190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.568083    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "kube-api-access-47ntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.570147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.570125    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token" pod "74dd4137-bcdc-4743-871b-1b9bd15ea461" (UID: "74dd4137-bcdc-4743-871b-1b9bd15ea461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:30.663472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663434    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663473    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663489    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dd4137-bcdc-4743-871b-1b9bd15ea461-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663503    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663516    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663529    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663541    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47ntr\" (UniqueName: \"kubernetes.io/projected/74dd4137-bcdc-4743-871b-1b9bd15ea461-kube-api-access-47ntr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.663847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.663554    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dd4137-bcdc-4743-871b-1b9bd15ea461-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:30.784539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.784500    4476 generic.go:356] "Generic (PLEG): container finished" podID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerID="73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db" exitCode=0
Nov 02 17:52:30.784774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.784558    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"
Nov 02 17:52:30.784774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.784685    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" event={"ID":"74dd4137-bcdc-4743-871b-1b9bd15ea461","Type":"ContainerDied","Data":"73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db"}
Nov 02 17:52:30.784774 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.784759    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z" event={"ID":"74dd4137-bcdc-4743-871b-1b9bd15ea461","Type":"ContainerDied","Data":"8d09834b5985cd1761b8b9225689630851fb2c00c53df0c69b85fc2eafd119fc"}
Nov 02 17:52:30.784952 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.784785    4476 scope.go:122] "RemoveContainer" containerID="73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db"
Nov 02 17:52:30.787986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.787365    4476 generic.go:356] "Generic (PLEG): container finished" podID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerID="40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1" exitCode=0
Nov 02 17:52:30.787986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.787473    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" event={"ID":"275e7402-8cf3-488c-95ed-e0da02d3dafe","Type":"ContainerDied","Data":"40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1"}
Nov 02 17:52:30.787986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.787526    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt" event={"ID":"275e7402-8cf3-488c-95ed-e0da02d3dafe","Type":"ContainerDied","Data":"396e43743f48bb5e0b8a1985b75aacfd16ef5eca8a72833b94ae1cdec59ecc3e"}
Nov 02 17:52:30.787986 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.787610    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"
Nov 02 17:52:30.798131 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.798090    4476 generic.go:356] "Generic (PLEG): container finished" podID="3910b518-983d-4b42-98c4-f3af2188d031" containerID="5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771" exitCode=0
Nov 02 17:52:30.798355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.798311    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" event={"ID":"3910b518-983d-4b42-98c4-f3af2188d031","Type":"ContainerDied","Data":"5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771"}
Nov 02 17:52:30.798461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.798368    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr" event={"ID":"3910b518-983d-4b42-98c4-f3af2188d031","Type":"ContainerDied","Data":"5ba79d3e40954b0f571df11280c7eeb8b3d357a237be1b23d73540dfaa13b699"}
Nov 02 17:52:30.798461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.798449    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"
Nov 02 17:52:30.819575 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.819548    4476 scope.go:122] "RemoveContainer" containerID="73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db"
Nov 02 17:52:30.819979 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:52:30.819955    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db\": container with ID starting with 73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db not found: ID does not exist" containerID="73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db"
Nov 02 17:52:30.820073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.819995    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db"} err="failed to get container status \"73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db\": rpc error: code = NotFound desc = could not find container \"73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db\": container with ID starting with 73b88f5f53b50f2f4508e4b039aee742214f701df106bd2d612cdd4a8edf32db not found: ID does not exist"
Nov 02 17:52:30.820073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.820026    4476 scope.go:122] "RemoveContainer" containerID="40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1"
Nov 02 17:52:30.832699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.832673    4476 scope.go:122] "RemoveContainer" containerID="40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1"
Nov 02 17:52:30.833342 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:52:30.833107    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1\": container with ID starting with 40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1 not found: ID does not exist" containerID="40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1"
Nov 02 17:52:30.833342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.833145    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1"} err="failed to get container status \"40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1\": rpc error: code = NotFound desc = could not find container \"40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1\": container with ID starting with 40b8b650485f0a89c0d9e02d068ed048c98b75a7f3383310e27a347e2e191ba1 not found: ID does not exist"
Nov 02 17:52:30.833342 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.833169    4476 scope.go:122] "RemoveContainer" containerID="5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771"
Nov 02 17:52:30.847749 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.847729    4476 scope.go:122] "RemoveContainer" containerID="5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771"
Nov 02 17:52:30.849959 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:52:30.849927    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771\": container with ID starting with 5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771 not found: ID does not exist" containerID="5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771"
Nov 02 17:52:30.850069 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:30.849970    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771"} err="failed to get container status \"5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771\": rpc error: code = NotFound desc = could not find container \"5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771\": container with ID starting with 5bc294a891ab19f79ce9612d00c984bb9216cf0037fba8bac76ef09d6a854771 not found: ID does not exist"
Nov 02 17:52:31.010478 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.010396    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"]
Nov 02 17:52:31.017705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.017669    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-94sqt"]
Nov 02 17:52:31.126677 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.126621    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"]
Nov 02 17:52:31.131055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.131021    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-nh7dr"]
Nov 02 17:52:31.238993 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.238944    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"]
Nov 02 17:52:31.242784 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.242751    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-g2c4z"]
Nov 02 17:52:31.385159 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.385122    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" path="/var/lib/kubelet/pods/275e7402-8cf3-488c-95ed-e0da02d3dafe/volumes"
Nov 02 17:52:31.385604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.385583    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3910b518-983d-4b42-98c4-f3af2188d031" path="/var/lib/kubelet/pods/3910b518-983d-4b42-98c4-f3af2188d031/volumes"
Nov 02 17:52:31.386026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.386012    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" path="/var/lib/kubelet/pods/74dd4137-bcdc-4743-871b-1b9bd15ea461/volumes"
Nov 02 17:52:31.621235 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.621195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:31.803151 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:31.803069    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:32.151990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.151947    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"]
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.152922    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.152948    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.152985    4476 cpu_manager.go:414] "containerMap: removing container" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.152992    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153015    4476 cpu_manager.go:414] "containerMap: removing container" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153022    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153060    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b16c95fe-e82f-480e-a274-ebf438f12ab0" containerName="payload-pre-processing"
Nov 02 17:52:32.153088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153069    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16c95fe-e82f-480e-a274-ebf438f12ab0" containerName="payload-pre-processing"
Nov 02 17:52:32.153571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153211    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="275e7402-8cf3-488c-95ed-e0da02d3dafe" containerName="istio-proxy"
Nov 02 17:52:32.153571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153223    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16c95fe-e82f-480e-a274-ebf438f12ab0" containerName="payload-pre-processing"
Nov 02 17:52:32.153571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153237    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3910b518-983d-4b42-98c4-f3af2188d031" containerName="istio-proxy"
Nov 02 17:52:32.153571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.153249    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dd4137-bcdc-4743-871b-1b9bd15ea461" containerName="istio-proxy"
Nov 02 17:52:32.180536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.180506    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.218705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.218554    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"]
Nov 02 17:52:32.258304 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.258235    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-666de0-dockercfg-lxqpt\""
Nov 02 17:52:32.258500 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.258476    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:32.258648 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.258616    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-666de0\""
Nov 02 17:52:32.285521 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.285474    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.285761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.285569    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxjk\" (UniqueName: \"kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk\") pod \"payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.387062 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.387024    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.387489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.387080    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxjk\" (UniqueName: \"kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk\") pod \"payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.387853 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.387832    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.444620 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.444538    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxjk\" (UniqueName: \"kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk\") pod \"payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.456680 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.456618    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:32.503071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.503027    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:32.894450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:32.894326    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"]
Nov 02 17:52:32.894614 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:52:32.894502    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e716ae_087a_44d7_8f34_c46ddf92787b.slice/crio-209de0a2b0c43f08dea8813850b099932b071d09bb34c1cbb27a7238dc757db6 WatchSource:0}: Error finding container 209de0a2b0c43f08dea8813850b099932b071d09bb34c1cbb27a7238dc757db6: Status 404 returned error can't find the container with id 209de0a2b0c43f08dea8813850b099932b071d09bb34c1cbb27a7238dc757db6
Nov 02 17:52:33.826574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:33.826531    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt" event={"ID":"02e716ae-087a-44d7-8f34-c46ddf92787b","Type":"ContainerStarted","Data":"9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c"}
Nov 02 17:52:33.827088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:33.826594    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt" event={"ID":"02e716ae-087a-44d7-8f34-c46ddf92787b","Type":"ContainerStarted","Data":"209de0a2b0c43f08dea8813850b099932b071d09bb34c1cbb27a7238dc757db6"}
Nov 02 17:52:33.827088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:33.826651    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:33.944610 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:33.944550    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt" podStartSLOduration=1.944532014 podStartE2EDuration="1.944532014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:33.939755306 +0000 UTC m=+1784.780361242" watchObservedRunningTime="2026-11-02 17:52:33.944532014 +0000 UTC m=+1784.785137949"
Nov 02 17:52:35.222453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:35.222410    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:35.408102 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:35.408073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:35.591797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:35.591713    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:38.916908 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:38.916872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:39.137971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.137937    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:39.584653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.577765    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"]
Nov 02 17:52:39.584653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.578044    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v" podUID="9260d9ed-3ae3-4562-84a8-35151a37bd62" containerName="maas-api" containerID="cri-o://fce3534e611bc3dd0c30799b2c006194980243f962f82d14c46001184dfcbc35" gracePeriod=30
Nov 02 17:52:39.736888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.736850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:39.918653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.917762    4476 generic.go:356] "Generic (PLEG): container finished" podID="9260d9ed-3ae3-4562-84a8-35151a37bd62" containerID="fce3534e611bc3dd0c30799b2c006194980243f962f82d14c46001184dfcbc35" exitCode=0
Nov 02 17:52:39.918653 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.917911    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v" event={"ID":"9260d9ed-3ae3-4562-84a8-35151a37bd62","Type":"ContainerDied","Data":"fce3534e611bc3dd0c30799b2c006194980243f962f82d14c46001184dfcbc35"}
Nov 02 17:52:39.993037 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:39.993009    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:52:40.037157 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.037117    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca\") pod \"9260d9ed-3ae3-4562-84a8-35151a37bd62\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") "
Nov 02 17:52:40.037476 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.037455    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs\") pod \"9260d9ed-3ae3-4562-84a8-35151a37bd62\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") "
Nov 02 17:52:40.040675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.037618    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4\" (UniqueName: \"kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4\") pod \"9260d9ed-3ae3-4562-84a8-35151a37bd62\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") "
Nov 02 17:52:40.040903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.040870    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca" pod "9260d9ed-3ae3-4562-84a8-35151a37bd62" (UID: "9260d9ed-3ae3-4562-84a8-35151a37bd62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:40.040980 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.040886    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\") pod \"9260d9ed-3ae3-4562-84a8-35151a37bd62\" (UID: \"9260d9ed-3ae3-4562-84a8-35151a37bd62\") "
Nov 02 17:52:40.041611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.041561    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9260d9ed-3ae3-4562-84a8-35151a37bd62-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:40.046522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.045279    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls" pod "9260d9ed-3ae3-4562-84a8-35151a37bd62" (UID: "9260d9ed-3ae3-4562-84a8-35151a37bd62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:52:40.048655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.047388    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs" pod "9260d9ed-3ae3-4562-84a8-35151a37bd62" (UID: "9260d9ed-3ae3-4562-84a8-35151a37bd62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:52:40.048655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.047558    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4" pod "9260d9ed-3ae3-4562-84a8-35151a37bd62" (UID: "9260d9ed-3ae3-4562-84a8-35151a37bd62"). InnerVolumeSpecName "kube-api-access-pdwg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:40.143816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.143784    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:40.143816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.143819    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdwg4\" (UniqueName: \"kubernetes.io/projected/9260d9ed-3ae3-4562-84a8-35151a37bd62-kube-api-access-pdwg4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:40.144083 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.143835    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9260d9ed-3ae3-4562-84a8-35151a37bd62-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:52:40.934444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.934388    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v" event={"ID":"9260d9ed-3ae3-4562-84a8-35151a37bd62","Type":"ContainerDied","Data":"ef61677e8d4734aad3c8f4ac156e8c0478e64c1f97b1451e8db05158c0eb111c"}
Nov 02 17:52:40.934990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.934532    4476 scope.go:122] "RemoveContainer" containerID="fce3534e611bc3dd0c30799b2c006194980243f962f82d14c46001184dfcbc35"
Nov 02 17:52:40.934990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:40.934788    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"
Nov 02 17:52:41.101437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:41.101403    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"]
Nov 02 17:52:41.150614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:41.150584    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e61288-545b69c84-v666v"]
Nov 02 17:52:41.384897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:41.384864    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9260d9ed-3ae3-4562-84a8-35151a37bd62" path="/var/lib/kubelet/pods/9260d9ed-3ae3-4562-84a8-35151a37bd62/volumes"
Nov 02 17:52:41.870572 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:41.870535    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:44.837372 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:44.837345    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:52:45.677024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:45.676961    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:45.872829 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:45.872797    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:46.398897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:46.398864    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:47.329544 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:47.329497    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:47.787190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:47.787158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:48.613039 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:48.612997    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:49.632557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:49.632516    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:52:49.645433 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:49.645408    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:52:49.944948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:49.944910    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:51.706888 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:51.706832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:52.520914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:52.520876    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:54.696871 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:54.696837    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:55.611218 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:55.611183    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:55.794522 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:55.794476    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:56.866456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:52:56.866397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:59.080258 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:52:59.080214    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:53:00.455857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:00.455819    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:01.574107 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:01.574049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:02.026235 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:02.026201    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:02.211216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:02.211184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:06.623318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:06.623261    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:11.652332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:11.652278    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:15.198894 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:15.198865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:15.790441 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:15.790406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:15.936230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:15.936193    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"]
Nov 02 17:53:15.936431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:15.936394    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy" containerID="cri-o://1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9" gracePeriod=30
Nov 02 17:53:18.264940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:18.264908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:18.585420 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:18.585294    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:21.215856 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.215792    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:53:21.248071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.248022    4476 generic.go:356] "Generic (PLEG): container finished" podID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerID="1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9" exitCode=0
Nov 02 17:53:21.248274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.248091    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"
Nov 02 17:53:21.248274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.248117    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" event={"ID":"f9898615-30ae-4d22-86e6-0f3d190e40b0","Type":"ContainerDied","Data":"1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9"}
Nov 02 17:53:21.248274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.248173    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t" event={"ID":"f9898615-30ae-4d22-86e6-0f3d190e40b0","Type":"ContainerDied","Data":"d718cdb4d7dbef63683c3d8c7cc4909d29765cc2bf2936492802b9c61fe42b8f"}
Nov 02 17:53:21.248274 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.248202    4476 scope.go:122] "RemoveContainer" containerID="1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9"
Nov 02 17:53:21.260969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.260944    4476 scope.go:122] "RemoveContainer" containerID="1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9"
Nov 02 17:53:21.261275 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:53:21.261251    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9\": container with ID starting with 1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9 not found: ID does not exist" containerID="1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9"
Nov 02 17:53:21.261352 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.261288    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9"} err="failed to get container status \"1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9\": rpc error: code = NotFound desc = could not find container \"1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9\": container with ID starting with 1e2a68c83306adc5e599ec0fe189e36a5059c35c08dbeae21665f75bf2d179e9 not found: ID does not exist"
Nov 02 17:53:21.316540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316506    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316555    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316673    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316760    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316788    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316815    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316852    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:21.316949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316908    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.316949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316937    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:21.317190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.316968    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.317190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317003    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert\") pod \"f9898615-30ae-4d22-86e6-0f3d190e40b0\" (UID: \"f9898615-30ae-4d22-86e6-0f3d190e40b0\") "
Nov 02 17:53:21.317190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317124    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:21.317190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317156    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:21.317543 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317518    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:21.317675 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317526    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.317909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317616    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.317909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317785    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.317909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.317799    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.319204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.319181    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:21.319346 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.319331    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:21.319488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.319466    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "kube-api-access-rkwkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:21.319488 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.319470    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy" pod "f9898615-30ae-4d22-86e6-0f3d190e40b0" (UID: "f9898615-30ae-4d22-86e6-0f3d190e40b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:21.418781 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.418749    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkwkd\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-kube-api-access-rkwkd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.418781 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.418779    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.418781 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.418789    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.419011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.418799    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9898615-30ae-4d22-86e6-0f3d190e40b0-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.419011 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.418807    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9898615-30ae-4d22-86e6-0f3d190e40b0-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:21.664418 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.664384    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"]
Nov 02 17:53:21.670474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:21.670438    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-m9c7t"]
Nov 02 17:53:23.371408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:23.371376    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:23.388029 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:23.387989    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" path="/var/lib/kubelet/pods/f9898615-30ae-4d22-86e6-0f3d190e40b0/volumes"
Nov 02 17:53:28.229529 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:28.229482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:28.429970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:28.429915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:28.616203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:28.616165    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:29.060351 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:53:29.060261    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:53:33.348207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:33.348035    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:39.406939 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.406907    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"]
Nov 02 17:53:39.407468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.407455    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy"
Nov 02 17:53:39.407513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.407469    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy"
Nov 02 17:53:39.407513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.407502    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9260d9ed-3ae3-4562-84a8-35151a37bd62" containerName="maas-api"
Nov 02 17:53:39.407513 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.407509    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9260d9ed-3ae3-4562-84a8-35151a37bd62" containerName="maas-api"
Nov 02 17:53:39.407608 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.407599    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9260d9ed-3ae3-4562-84a8-35151a37bd62" containerName="maas-api"
Nov 02 17:53:39.407686 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.407613    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9898615-30ae-4d22-86e6-0f3d190e40b0" containerName="istio-proxy"
Nov 02 17:53:39.412411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.412390    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.513955 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.513923    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5bf666-serving-cert\""
Nov 02 17:53:39.518391 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.518349    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"]
Nov 02 17:53:39.573614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.573571    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.573843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.573660    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skb9r\" (UniqueName: \"kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.573843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.573723    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.573843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.573804    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.666692 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.666579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:39.675677 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.675616    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.675961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.675938    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skb9r\" (UniqueName: \"kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.676115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.676100    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.676272 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.676259    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.678065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.677281    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.685618 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.683255    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.685765 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:53:39.683709    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-serving-cert: secret "maas-api-e2e-api-a-5bf666-serving-cert" not found
Nov 02 17:53:39.685921 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:53:39.685909    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls podName:9e02d380-db11-467f-8d91-f28677dcf9aa nodeName:}" failed. No retries permitted until 2026-11-02 17:53:40.18588464 +0000 UTC m=+1851.026490556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls") pod "maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" (UID: "9e02d380-db11-467f-8d91-f28677dcf9aa") : secret "maas-api-e2e-api-a-5bf666-serving-cert" not found
Nov 02 17:53:39.742412 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.742371    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skb9r\" (UniqueName: \"kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:39.895901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:39.895866    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:40.081251 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:40.081151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:40.285143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:40.285107    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:40.288007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:40.287971    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\") pod \"maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:40.326284 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:40.326242    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:40.714431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:40.714404    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"]
Nov 02 17:53:40.715406 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:53:40.715378    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e02d380_db11_467f_8d91_f28677dcf9aa.slice/crio-3586d1013b8e035f395804a1670e407eba0d83732440af2e642a5037d9246e4d WatchSource:0}: Error finding container 3586d1013b8e035f395804a1670e407eba0d83732440af2e642a5037d9246e4d: Status 404 returned error can't find the container with id 3586d1013b8e035f395804a1670e407eba0d83732440af2e642a5037d9246e4d
Nov 02 17:53:40.717004 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:40.716986    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:53:41.420298 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:41.420259    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" event={"ID":"9e02d380-db11-467f-8d91-f28677dcf9aa","Type":"ContainerStarted","Data":"3586d1013b8e035f395804a1670e407eba0d83732440af2e642a5037d9246e4d"}
Nov 02 17:53:42.458204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:42.458161    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" event={"ID":"9e02d380-db11-467f-8d91-f28677dcf9aa","Type":"ContainerStarted","Data":"a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21"}
Nov 02 17:53:42.578126 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:42.578074    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" podStartSLOduration=2.550730815 podStartE2EDuration="3.578059229s" totalImagesPullingTime="1.027328414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:42.576271815 +0000 UTC m=+1853.416877749" watchObservedRunningTime="2026-11-02 17:53:42.578059229 +0000 UTC m=+1853.418665215"
Nov 02 17:53:43.464937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:43.464905    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:44.686697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:44.686593    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:49.477795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:49.477758    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:53:49.656196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:49.656156    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:50.280698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:50.280598    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:53.337275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:53.337241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:53.461381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:53.461347    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"]
Nov 02 17:53:53.570203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:53.570153    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"]
Nov 02 17:53:53.913983 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:53.913933    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:54.115844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:54.115801    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:54.838673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:54.838582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:54.978366 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:54.978333    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"]
Nov 02 17:53:54.978605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:54.978573    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" containerID="cri-o://9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7" gracePeriod=30
Nov 02 17:53:57.521472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:57.521434    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"]
Nov 02 17:53:57.522071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:57.522041    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy" containerID="cri-o://532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113" gracePeriod=30
Nov 02 17:53:58.462387 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.462338    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls" podUID="962e1b3b-adb7-464f-bac1-cfd66c443195" containerName="payload-processing" containerID="cri-o://f9d0aec37739d80c3044e76f0a717e7b646273386f8bd3927f682b4c16a87144" gracePeriod=25
Nov 02 17:53:58.570917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.570852    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc" podUID="14fea00c-064b-48ed-a8aa-612704e68898" containerName="payload-pre-processing" containerID="cri-o://b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818" gracePeriod=25
Nov 02 17:53:58.585316 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.585279    4476 generic.go:356] "Generic (PLEG): container finished" podID="962e1b3b-adb7-464f-bac1-cfd66c443195" containerID="f9d0aec37739d80c3044e76f0a717e7b646273386f8bd3927f682b4c16a87144" exitCode=0
Nov 02 17:53:58.585481 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.585354    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls" event={"ID":"962e1b3b-adb7-464f-bac1-cfd66c443195","Type":"ContainerDied","Data":"f9d0aec37739d80c3044e76f0a717e7b646273386f8bd3927f682b4c16a87144"}
Nov 02 17:53:58.761216 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.760970    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:53:58.837649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.837592    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5\" (UniqueName: \"kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5\") pod \"962e1b3b-adb7-464f-bac1-cfd66c443195\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") "
Nov 02 17:53:58.837911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.837895    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume\") pod \"962e1b3b-adb7-464f-bac1-cfd66c443195\" (UID: \"962e1b3b-adb7-464f-bac1-cfd66c443195\") "
Nov 02 17:53:58.838294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.838271    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume" pod "962e1b3b-adb7-464f-bac1-cfd66c443195" (UID: "962e1b3b-adb7-464f-bac1-cfd66c443195"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:58.840025 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.840002    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5" pod "962e1b3b-adb7-464f-bac1-cfd66c443195" (UID: "962e1b3b-adb7-464f-bac1-cfd66c443195"). InnerVolumeSpecName "kube-api-access-sktd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:58.860196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.860169    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:53:58.892018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.891825    4476 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:53:58.892018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.891912    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:53:58.939758 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.939727    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume\") pod \"14fea00c-064b-48ed-a8aa-612704e68898\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") "
Nov 02 17:53:58.939988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.939807    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz\" (UniqueName: \"kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz\") pod \"14fea00c-064b-48ed-a8aa-612704e68898\" (UID: \"14fea00c-064b-48ed-a8aa-612704e68898\") "
Nov 02 17:53:58.940273 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.940239    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume" pod "14fea00c-064b-48ed-a8aa-612704e68898" (UID: "14fea00c-064b-48ed-a8aa-612704e68898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:58.940273 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.940261    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e1b3b-adb7-464f-bac1-cfd66c443195-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:58.940273 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.940277    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktd5\" (UniqueName: \"kubernetes.io/projected/962e1b3b-adb7-464f-bac1-cfd66c443195-kube-api-access-sktd5\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:58.942414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:58.942369    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz" pod "14fea00c-064b-48ed-a8aa-612704e68898" (UID: "14fea00c-064b-48ed-a8aa-612704e68898"). InnerVolumeSpecName "kube-api-access-cwrdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:59.042050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.041959    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fea00c-064b-48ed-a8aa-612704e68898-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:59.042050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.042000    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrdz\" (UniqueName: \"kubernetes.io/projected/14fea00c-064b-48ed-a8aa-612704e68898-kube-api-access-cwrdz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:53:59.081752 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:53:59.081716    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:53:59.176812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.176779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:59.593870 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.593828    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls" event={"ID":"962e1b3b-adb7-464f-bac1-cfd66c443195","Type":"ContainerDied","Data":"08515f74b9bb2ecbb634692b83a84dd1c10ad90594573d3fe94fad86a177581c"}
Nov 02 17:53:59.594385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.593909    4476 scope.go:122] "RemoveContainer" containerID="f9d0aec37739d80c3044e76f0a717e7b646273386f8bd3927f682b4c16a87144"
Nov 02 17:53:59.594385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.593839    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"
Nov 02 17:53:59.595574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.595552    4476 generic.go:356] "Generic (PLEG): container finished" podID="14fea00c-064b-48ed-a8aa-612704e68898" containerID="b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818" exitCode=0
Nov 02 17:53:59.596468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.595594    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc" event={"ID":"14fea00c-064b-48ed-a8aa-612704e68898","Type":"ContainerDied","Data":"b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818"}
Nov 02 17:53:59.596468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.595617    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"
Nov 02 17:53:59.596468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.595671    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc" event={"ID":"14fea00c-064b-48ed-a8aa-612704e68898","Type":"ContainerDied","Data":"553b5d8297481b0f47b68378f44743dc494a897b046ee75ae0889115e3c985bb"}
Nov 02 17:53:59.604797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.604778    4476 scope.go:122] "RemoveContainer" containerID="b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818"
Nov 02 17:53:59.614704 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.614686    4476 scope.go:122] "RemoveContainer" containerID="b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818"
Nov 02 17:53:59.615017 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:53:59.614993    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818\": container with ID starting with b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818 not found: ID does not exist" containerID="b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818"
Nov 02 17:53:59.615075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.615026    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818"} err="failed to get container status \"b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818\": rpc error: code = NotFound desc = could not find container \"b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818\": container with ID starting with b6f5649470c4f583649d162009d8b7a2a0968952429c3aa7515353b0d220b818 not found: ID does not exist"
Nov 02 17:53:59.760611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.760517    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"]
Nov 02 17:53:59.809315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.809282    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-6676d88bb7-zpczc"]
Nov 02 17:53:59.873776 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.873717    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"]
Nov 02 17:53:59.879381 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:53:59.879347    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-64f654f5d7-q9hls"]
Nov 02 17:54:00.299769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.299740    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:54:00.462983 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.462945    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463039    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463072    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463105    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463189    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463247    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463234    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:00.463422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463293    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463359    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.463422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.463395    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket\") pod \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\" (UID: \"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc\") "
Nov 02 17:54:00.464247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.464105    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.464247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.464219    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:00.464917 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.464695    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:00.465271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.465243    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:00.465378 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.465313    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:00.466267 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.466244    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:00.466663 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.466595    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:00.466811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.466789    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:00.467612 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.467592    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8" pod "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" (UID: "50ce98c5-ae32-46e6-a0e8-d40f6621c4dc"). InnerVolumeSpecName "kube-api-access-hssd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.565928    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.565961    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.565972    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.565981    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hssd8\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-kube-api-access-hssd8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.565991    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.566000    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.566008    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.566015 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.566018    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.605699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.605662    4476 generic.go:356] "Generic (PLEG): container finished" podID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerID="9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7" exitCode=0
Nov 02 17:54:00.606196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.605725    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"
Nov 02 17:54:00.606196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.605756    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" event={"ID":"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc","Type":"ContainerDied","Data":"9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7"}
Nov 02 17:54:00.606196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.605826    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw" event={"ID":"50ce98c5-ae32-46e6-a0e8-d40f6621c4dc","Type":"ContainerDied","Data":"465b57c3da1dae5e517b6ae398f99bde0b4ed4bb5a43f14bf6c829cf6093be1b"}
Nov 02 17:54:00.606196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.605858    4476 scope.go:122] "RemoveContainer" containerID="9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7"
Nov 02 17:54:00.620772 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.620751    4476 scope.go:122] "RemoveContainer" containerID="9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7"
Nov 02 17:54:00.621142 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:00.621119    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7\": container with ID starting with 9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7 not found: ID does not exist" containerID="9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7"
Nov 02 17:54:00.621215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.621156    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7"} err="failed to get container status \"9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7\": rpc error: code = NotFound desc = could not find container \"9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7\": container with ID starting with 9465f6fdb3a4e277b9ee56bac54ede23d0b828f9aca88e3ebce860b7644c14b7 not found: ID does not exist"
Nov 02 17:54:00.770124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.770088    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"]
Nov 02 17:54:00.774194 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:00.774164    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-fcktw"]
Nov 02 17:54:01.008008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:01.007915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:01.383437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:01.383404    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fea00c-064b-48ed-a8aa-612704e68898" path="/var/lib/kubelet/pods/14fea00c-064b-48ed-a8aa-612704e68898/volumes"
Nov 02 17:54:01.383926 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:01.383911    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" path="/var/lib/kubelet/pods/50ce98c5-ae32-46e6-a0e8-d40f6621c4dc/volumes"
Nov 02 17:54:01.384279 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:01.384266    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962e1b3b-adb7-464f-bac1-cfd66c443195" path="/var/lib/kubelet/pods/962e1b3b-adb7-464f-bac1-cfd66c443195/volumes"
Nov 02 17:54:02.807097 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.807072    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:54:02.897604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897572    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.897604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897610    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.897900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897667    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.897900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897719    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.897900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897808    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.897900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897861    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.898109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897945    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:02.898109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897963    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:02.898109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897979    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.898109 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.897997    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:02.898303 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898110    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.898303 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898131    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:02.898303 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898207    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token\") pod \"821b43c6-579c-4af9-9328-1d3d113830ee\" (UID: \"821b43c6-579c-4af9-9328-1d3d113830ee\") "
Nov 02 17:54:02.898303 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898247    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:02.898816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898745    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:02.898816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898768    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b43c6-579c-4af9-9328-1d3d113830ee-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:02.898816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898786    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:02.898816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898801    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:02.898816 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.898816    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:02.900605 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.900585    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "kube-api-access-svq4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:02.900727 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.900621    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:02.900727 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.900674    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:02.900824 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.900802    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token" pod "821b43c6-579c-4af9-9328-1d3d113830ee" (UID: "821b43c6-579c-4af9-9328-1d3d113830ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:03.000008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:02.999975    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b43c6-579c-4af9-9328-1d3d113830ee-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:03.000189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.000011    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:03.000189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.000028    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svq4h\" (UniqueName: \"kubernetes.io/projected/821b43c6-579c-4af9-9328-1d3d113830ee-kube-api-access-svq4h\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:03.000189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.000045    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b43c6-579c-4af9-9328-1d3d113830ee-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:03.081065 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.080982    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"]
Nov 02 17:54:03.081300 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.081222    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" containerID="cri-o://c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6" gracePeriod=30
Nov 02 17:54:03.085089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.084717    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"]
Nov 02 17:54:03.085089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.084904    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" containerID="cri-o://8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a" gracePeriod=30
Nov 02 17:54:03.635862 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.635829    4476 generic.go:356] "Generic (PLEG): container finished" podID="821b43c6-579c-4af9-9328-1d3d113830ee" containerID="532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113" exitCode=0
Nov 02 17:54:03.636031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.635899    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"
Nov 02 17:54:03.636031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.635917    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" event={"ID":"821b43c6-579c-4af9-9328-1d3d113830ee","Type":"ContainerDied","Data":"532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113"}
Nov 02 17:54:03.636031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.635985    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn" event={"ID":"821b43c6-579c-4af9-9328-1d3d113830ee","Type":"ContainerDied","Data":"c8259e90aef0ba984d9c51b963ca358f4540cb0af50805d7b8a672a7ffe3d1f0"}
Nov 02 17:54:03.636031 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.636011    4476 scope.go:122] "RemoveContainer" containerID="532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113"
Nov 02 17:54:03.646940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.646718    4476 scope.go:122] "RemoveContainer" containerID="532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113"
Nov 02 17:54:03.647057 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:03.647029    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113\": container with ID starting with 532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113 not found: ID does not exist" containerID="532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113"
Nov 02 17:54:03.647112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.647066    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113"} err="failed to get container status \"532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113\": rpc error: code = NotFound desc = could not find container \"532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113\": container with ID starting with 532f86c47447fef3b71644bd8a540d0923869f3fbe76c6e0a2d4a90b60a5d113 not found: ID does not exist"
Nov 02 17:54:03.802443 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.802410    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"]
Nov 02 17:54:03.857349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:03.856134    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-phpkn"]
Nov 02 17:54:05.386585 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:05.386550    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" path="/var/lib/kubelet/pods/821b43c6-579c-4af9-9328-1d3d113830ee/volumes"
Nov 02 17:54:05.845297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:05.845205    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:06.801532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:06.801491    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:54:06.801942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:06.801547    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:54:07.475394 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:07.475348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:07.507267 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:07.507226    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:54:07.507447 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:07.507309    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:54:07.681304 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:07.681269    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:08.376425 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.376355    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:54:08.413825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.413799    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:54:08.467203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467168    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467205    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467232    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467268    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467323    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467354    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467385    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467425    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.467457 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467454    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467497    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467502    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467511    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467540    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467570    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467620    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467672    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.467759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467723    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.468311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467770    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.468311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467828    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.468311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.467919    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.468311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.468084    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.468311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.468121    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:08.468311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.468306    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.468659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.468360    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.468659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.468409    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert\") pod \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\" (UID: \"b14c7ef8-a016-4ab1-bbae-b1ade638857b\") "
Nov 02 17:54:08.468659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.468438    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy\") pod \"894105b3-ee9b-48e1-979e-f57fb8748065\" (UID: \"894105b3-ee9b-48e1-979e-f57fb8748065\") "
Nov 02 17:54:08.469100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469052    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.469100 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469083    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:08.469588 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469562    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894105b3-ee9b-48e1-979e-f57fb8748065-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469592    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469608    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469622    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469677    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469691    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469703    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.469718 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469716    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.470089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469732    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.470089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.469747    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.470435 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.470408    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "kube-api-access-4g78q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:08.471013 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.470976    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:08.471121 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.471070    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:08.471984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.471943    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "kube-api-access-tjkbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:08.472122 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.472098    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:08.472852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.472828    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.473539 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.473514    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token" pod "894105b3-ee9b-48e1-979e-f57fb8748065" (UID: "894105b3-ee9b-48e1-979e-f57fb8748065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:08.473698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.473674    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy" pod "b14c7ef8-a016-4ab1-bbae-b1ade638857b" (UID: "b14c7ef8-a016-4ab1-bbae-b1ade638857b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:08.570638 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570588    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkbw\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-kube-api-access-tjkbw\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570656    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g78q\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-kube-api-access-4g78q\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570677    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894105b3-ee9b-48e1-979e-f57fb8748065-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570691    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570705    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570718    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14c7ef8-a016-4ab1-bbae-b1ade638857b-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570734    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894105b3-ee9b-48e1-979e-f57fb8748065-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.570814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.570749    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894105b3-ee9b-48e1-979e-f57fb8748065-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:08.696422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.696379    4476 generic.go:356] "Generic (PLEG): container finished" podID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerID="8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a" exitCode=0
Nov 02 17:54:08.696621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.696446    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"
Nov 02 17:54:08.696621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.696472    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" event={"ID":"b14c7ef8-a016-4ab1-bbae-b1ade638857b","Type":"ContainerDied","Data":"8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a"}
Nov 02 17:54:08.696621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.696551    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd" event={"ID":"b14c7ef8-a016-4ab1-bbae-b1ade638857b","Type":"ContainerDied","Data":"85e6a6da5075074be047d32e6509f3806e0d8459f3a026f886e99bc667f134f3"}
Nov 02 17:54:08.696621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.696581    4476 scope.go:122] "RemoveContainer" containerID="8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a"
Nov 02 17:54:08.698591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.698568    4476 generic.go:356] "Generic (PLEG): container finished" podID="894105b3-ee9b-48e1-979e-f57fb8748065" containerID="c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6" exitCode=0
Nov 02 17:54:08.698728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.698687    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" event={"ID":"894105b3-ee9b-48e1-979e-f57fb8748065","Type":"ContainerDied","Data":"c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6"}
Nov 02 17:54:08.698782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.698726    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4" event={"ID":"894105b3-ee9b-48e1-979e-f57fb8748065","Type":"ContainerDied","Data":"d3151dd1e775b77308263a38fdbf221fde40ae23abefaa3cbe1097a9f7dc6469"}
Nov 02 17:54:08.698782 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.698727    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"
Nov 02 17:54:08.712030 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.712012    4476 scope.go:122] "RemoveContainer" containerID="8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a"
Nov 02 17:54:08.712357 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:08.712336    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a\": container with ID starting with 8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a not found: ID does not exist" containerID="8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a"
Nov 02 17:54:08.712442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.712365    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a"} err="failed to get container status \"8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a\": rpc error: code = NotFound desc = could not find container \"8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a\": container with ID starting with 8a3a111ecd2c07805bf0b002cbc16991076be4e47337dd60e3fdc6cb2e7bb05a not found: ID does not exist"
Nov 02 17:54:08.712442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.712382    4476 scope.go:122] "RemoveContainer" containerID="c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6"
Nov 02 17:54:08.721982 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.721965    4476 scope.go:122] "RemoveContainer" containerID="c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6"
Nov 02 17:54:08.722285 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:08.722265    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6\": container with ID starting with c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6 not found: ID does not exist" containerID="c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6"
Nov 02 17:54:08.722380 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.722289    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6"} err="failed to get container status \"c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6\": rpc error: code = NotFound desc = could not find container \"c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6\": container with ID starting with c9f1ab3f86a6544db4771bfcf686ffc42280ac31cfce4285542b4bd2702a5be6 not found: ID does not exist"
Nov 02 17:54:08.862032 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.861993    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"]
Nov 02 17:54:08.868869 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.868838    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fx7nd"]
Nov 02 17:54:08.976990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.976906    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"]
Nov 02 17:54:08.981435 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:08.981399    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-shmz4"]
Nov 02 17:54:09.192945 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:09.192916    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:09.386199 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:09.386158    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" path="/var/lib/kubelet/pods/894105b3-ee9b-48e1-979e-f57fb8748065/volumes"
Nov 02 17:54:09.386794 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:09.386777    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" path="/var/lib/kubelet/pods/b14c7ef8-a016-4ab1-bbae-b1ade638857b/volumes"
Nov 02 17:54:09.723382 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:09.723271    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:10.931349 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:10.931318    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:12.550890 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:12.550856    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:13.554135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554096    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"]
Nov 02 17:54:13.554900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554882    4476 cpu_manager.go:414] "containerMap: removing container" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554901    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554926    4476 cpu_manager.go:414] "containerMap: removing container" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554931    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554938    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554942    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554952    4476 cpu_manager.go:414] "containerMap: removing container" podUID="14fea00c-064b-48ed-a8aa-612704e68898" containerName="payload-pre-processing"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554957    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fea00c-064b-48ed-a8aa-612704e68898" containerName="payload-pre-processing"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554969    4476 cpu_manager.go:414] "containerMap: removing container" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554973    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554982    4476 cpu_manager.go:414] "containerMap: removing container" podUID="962e1b3b-adb7-464f-bac1-cfd66c443195" containerName="payload-processing"
Nov 02 17:54:13.554996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.554986    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962e1b3b-adb7-464f-bac1-cfd66c443195" containerName="payload-processing"
Nov 02 17:54:13.555467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.555058    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="821b43c6-579c-4af9-9328-1d3d113830ee" containerName="istio-proxy"
Nov 02 17:54:13.555467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.555066    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="962e1b3b-adb7-464f-bac1-cfd66c443195" containerName="payload-processing"
Nov 02 17:54:13.555467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.555075    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="894105b3-ee9b-48e1-979e-f57fb8748065" containerName="istio-proxy"
Nov 02 17:54:13.555467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.555084    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ce98c5-ae32-46e6-a0e8-d40f6621c4dc" containerName="istio-proxy"
Nov 02 17:54:13.555467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.555089    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14c7ef8-a016-4ab1-bbae-b1ade638857b" containerName="istio-proxy"
Nov 02 17:54:13.555467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.555095    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fea00c-064b-48ed-a8aa-612704e68898" containerName="payload-pre-processing"
Nov 02 17:54:13.558567 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.558546    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.627740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.627705    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ccf013-openshift-default-dockercfg-zfxsb\""
Nov 02 17:54:13.643348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643312    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643543 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643377    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643543 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643503    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643703 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643543    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643703 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643668    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643711    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643810 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643779    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643839    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.643983 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.643958    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsv4\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.676932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.676894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:13.677416 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.677390    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"]
Nov 02 17:54:13.745377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745344    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745418    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745456    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745479    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745517    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745600 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745564    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745609    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745685    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.745923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.745762    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsv4\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.746817 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.746786    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.747106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.747079    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.747200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.747130    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.747579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.747557    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.747873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.747848    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.751665 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.748873    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.751665 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.749549    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.805330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.805250    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsv4\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.805545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.805527    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token\") pod \"e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") " pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.873838 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.873803    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:13.984609 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:13.984462    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:14.309611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.309580    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"]
Nov 02 17:54:14.310856 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:54:14.310829    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0c4a68_ea6a_43ea_958a_87be9d86be8f.slice/crio-1c980f002c375b48bf2fa21cf7bb8a1e058307d6d03405427ea89318ae0d39b0 WatchSource:0}: Error finding container 1c980f002c375b48bf2fa21cf7bb8a1e058307d6d03405427ea89318ae0d39b0: Status 404 returned error can't find the container with id 1c980f002c375b48bf2fa21cf7bb8a1e058307d6d03405427ea89318ae0d39b0
Nov 02 17:54:14.313438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.313406    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:54:14.313506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.313476    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:54:14.313543 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.313504    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:54:14.748922 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.748886    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" event={"ID":"ab0c4a68-ea6a-43ea-958a-87be9d86be8f","Type":"ContainerStarted","Data":"2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"}
Nov 02 17:54:14.749317 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.748949    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" event={"ID":"ab0c4a68-ea6a-43ea-958a-87be9d86be8f","Type":"ContainerStarted","Data":"1c980f002c375b48bf2fa21cf7bb8a1e058307d6d03405427ea89318ae0d39b0"}
Nov 02 17:54:14.868377 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.868312    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podStartSLOduration=1.868294136 podStartE2EDuration="1.868294136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:14.865761508 +0000 UTC m=+1885.706367443" watchObservedRunningTime="2026-11-02 17:54:14.868294136 +0000 UTC m=+1885.708900073"
Nov 02 17:54:14.874245 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:14.874220    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:15.880840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:15.880804    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:16.397320 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:16.397290    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"]
Nov 02 17:54:16.532561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:16.532525    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"]
Nov 02 17:54:16.626213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:16.626182    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:16.769697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:16.769142    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:16.773690 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:16.770760    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:54:16.817668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:16.816494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:17.007180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:17.007142    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:17.252010 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:17.251970    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:17.758831 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:17.758788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:17.983407 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:17.983368    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:18.307226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:18.307191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:21.398184 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.398141    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w" podUID="92aac989-f02c-449e-aa23-ecef70245d73" containerName="payload-processing" containerID="cri-o://5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a" gracePeriod=25
Nov 02 17:54:21.533305 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.533261    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g" podUID="161cf413-1264-451a-8085-0226d0ef3ca0" containerName="payload-pre-processing" containerID="cri-o://14cec23926e08f9802e1340ee802cfc7f69a41baf8ae89463649c62d0a3900f0" gracePeriod=25
Nov 02 17:54:21.694885 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.694854    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:54:21.749164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.749126    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume\") pod \"92aac989-f02c-449e-aa23-ecef70245d73\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") "
Nov 02 17:54:21.749332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.749282    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h\" (UniqueName: \"kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h\") pod \"92aac989-f02c-449e-aa23-ecef70245d73\" (UID: \"92aac989-f02c-449e-aa23-ecef70245d73\") "
Nov 02 17:54:21.750222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.750197    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume" pod "92aac989-f02c-449e-aa23-ecef70245d73" (UID: "92aac989-f02c-449e-aa23-ecef70245d73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:21.754337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.754307    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h" pod "92aac989-f02c-449e-aa23-ecef70245d73" (UID: "92aac989-f02c-449e-aa23-ecef70245d73"). InnerVolumeSpecName "kube-api-access-t6m8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:21.769246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.768760    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:21.862723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.860331    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92aac989-f02c-449e-aa23-ecef70245d73-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:21.862723 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.860368    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6m8h\" (UniqueName: \"kubernetes.io/projected/92aac989-f02c-449e-aa23-ecef70245d73-kube-api-access-t6m8h\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:21.865884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.864730    4476 generic.go:356] "Generic (PLEG): container finished" podID="92aac989-f02c-449e-aa23-ecef70245d73" containerID="5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a" exitCode=0
Nov 02 17:54:21.865884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.864860    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w" event={"ID":"92aac989-f02c-449e-aa23-ecef70245d73","Type":"ContainerDied","Data":"5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a"}
Nov 02 17:54:21.865884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.864868    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"
Nov 02 17:54:21.865884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.864946    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w" event={"ID":"92aac989-f02c-449e-aa23-ecef70245d73","Type":"ContainerDied","Data":"ee365254c6d364786cf31d12953e316d8430a27c20463f7427c288e3d865e26a"}
Nov 02 17:54:21.865884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.864981    4476 scope.go:122] "RemoveContainer" containerID="5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a"
Nov 02 17:54:21.873923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.873196    4476 generic.go:356] "Generic (PLEG): container finished" podID="161cf413-1264-451a-8085-0226d0ef3ca0" containerID="14cec23926e08f9802e1340ee802cfc7f69a41baf8ae89463649c62d0a3900f0" exitCode=0
Nov 02 17:54:21.873923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.873420    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g" event={"ID":"161cf413-1264-451a-8085-0226d0ef3ca0","Type":"ContainerDied","Data":"14cec23926e08f9802e1340ee802cfc7f69a41baf8ae89463649c62d0a3900f0"}
Nov 02 17:54:21.881230 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.881142    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:54:21.896968 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.896804    4476 scope.go:122] "RemoveContainer" containerID="5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a"
Nov 02 17:54:21.897411 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:21.897386    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a\": container with ID starting with 5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a not found: ID does not exist" containerID="5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a"
Nov 02 17:54:21.897525 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.897422    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a"} err="failed to get container status \"5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a\": rpc error: code = NotFound desc = could not find container \"5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a\": container with ID starting with 5c76098504ec3b2210656e283e263deb3a0b5c56acc505f3ab4527052002185a not found: ID does not exist"
Nov 02 17:54:21.962026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.961935    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf\" (UniqueName: \"kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf\") pod \"161cf413-1264-451a-8085-0226d0ef3ca0\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") "
Nov 02 17:54:21.962026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.962019    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume\") pod \"161cf413-1264-451a-8085-0226d0ef3ca0\" (UID: \"161cf413-1264-451a-8085-0226d0ef3ca0\") "
Nov 02 17:54:21.962378 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.962355    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume" pod "161cf413-1264-451a-8085-0226d0ef3ca0" (UID: "161cf413-1264-451a-8085-0226d0ef3ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:21.962489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.962475    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161cf413-1264-451a-8085-0226d0ef3ca0-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:21.964254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:21.964229    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf" pod "161cf413-1264-451a-8085-0226d0ef3ca0" (UID: "161cf413-1264-451a-8085-0226d0ef3ca0"). InnerVolumeSpecName "kube-api-access-6bmqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:22.019486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.019452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:22.081654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.081336    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmqf\" (UniqueName: \"kubernetes.io/projected/161cf413-1264-451a-8085-0226d0ef3ca0-kube-api-access-6bmqf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:22.144060 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144025    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"]
Nov 02 17:54:22.144614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144598    4476 cpu_manager.go:414] "containerMap: removing container" podUID="92aac989-f02c-449e-aa23-ecef70245d73" containerName="payload-processing"
Nov 02 17:54:22.144713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144616    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92aac989-f02c-449e-aa23-ecef70245d73" containerName="payload-processing"
Nov 02 17:54:22.144713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144658    4476 cpu_manager.go:414] "containerMap: removing container" podUID="161cf413-1264-451a-8085-0226d0ef3ca0" containerName="payload-pre-processing"
Nov 02 17:54:22.144713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144664    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161cf413-1264-451a-8085-0226d0ef3ca0" containerName="payload-pre-processing"
Nov 02 17:54:22.144818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144753    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="161cf413-1264-451a-8085-0226d0ef3ca0" containerName="payload-pre-processing"
Nov 02 17:54:22.144818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.144765    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="92aac989-f02c-449e-aa23-ecef70245d73" containerName="payload-processing"
Nov 02 17:54:22.149153 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.149138    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.165211 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.165170    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"]
Nov 02 17:54:22.170576 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.170548    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-6bf99d8df7-cnb2w"]
Nov 02 17:54:22.201593 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.201551    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ccf013\""
Nov 02 17:54:22.201876 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.201856    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ccf013-dockercfg-vvzx4\""
Nov 02 17:54:22.204900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.204879    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"]
Nov 02 17:54:22.291714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.288383    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.291714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.288495    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgzk\" (UniqueName: \"kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.324654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.323685    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"]
Nov 02 17:54:22.346542 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.342510    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.389992 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.389959    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"]
Nov 02 17:54:22.390233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.390210    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.390329 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.390310    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgzk\" (UniqueName: \"kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.391158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.391135    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.448752 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.448720    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgzk\" (UniqueName: \"kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.468384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.468349    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:22.491366 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.491329    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmhp\" (UniqueName: \"kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp\") pod \"payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.491499 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.491463    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.592868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.592769    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.593042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.592881    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmhp\" (UniqueName: \"kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp\") pod \"payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.593664 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.593603    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.650415 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.650382    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmhp\" (UniqueName: \"kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp\") pod \"payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.686954 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.686923    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:22.850435 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.850404    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"]
Nov 02 17:54:22.852452 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:54:22.852415    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ae4f0e_5ec5_4194_9971_5e8d04955e22.slice/crio-3d866966356970da3e74660e3ce50ae59f5be06f063baf31b91a42c50c298889 WatchSource:0}: Error finding container 3d866966356970da3e74660e3ce50ae59f5be06f063baf31b91a42c50c298889: Status 404 returned error can't find the container with id 3d866966356970da3e74660e3ce50ae59f5be06f063baf31b91a42c50c298889
Nov 02 17:54:22.884867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.884694    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g" event={"ID":"161cf413-1264-451a-8085-0226d0ef3ca0","Type":"ContainerDied","Data":"956f2ee9288f3db4f1e55c5f9e03e4ee27b0a78c59c0a8e58418dcd4eb397145"}
Nov 02 17:54:22.884867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.884766    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"
Nov 02 17:54:22.884867 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.884773    4476 scope.go:122] "RemoveContainer" containerID="14cec23926e08f9802e1340ee802cfc7f69a41baf8ae89463649c62d0a3900f0"
Nov 02 17:54:22.889912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:22.889878    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw" event={"ID":"e8ae4f0e-5ec5-4194-9971-5e8d04955e22","Type":"ContainerStarted","Data":"3d866966356970da3e74660e3ce50ae59f5be06f063baf31b91a42c50c298889"}
Nov 02 17:54:23.054998 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.054963    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"]
Nov 02 17:54:23.066817 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.066779    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-5fd9c6fb6c-84p8g"]
Nov 02 17:54:23.078516 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.078488    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"]
Nov 02 17:54:23.080576 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:54:23.080550    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11f1e3c_d8be_4fbe_80e3_64496f40683c.slice/crio-dd1485ca52ee8cffb9fad9b584626bd4b38132482b98e1db987e8410e14b1daa WatchSource:0}: Error finding container dd1485ca52ee8cffb9fad9b584626bd4b38132482b98e1db987e8410e14b1daa: Status 404 returned error can't find the container with id dd1485ca52ee8cffb9fad9b584626bd4b38132482b98e1db987e8410e14b1daa
Nov 02 17:54:23.391308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.391261    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161cf413-1264-451a-8085-0226d0ef3ca0" path="/var/lib/kubelet/pods/161cf413-1264-451a-8085-0226d0ef3ca0/volumes"
Nov 02 17:54:23.391970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.391946    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92aac989-f02c-449e-aa23-ecef70245d73" path="/var/lib/kubelet/pods/92aac989-f02c-449e-aa23-ecef70245d73/volumes"
Nov 02 17:54:23.672852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.672757    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:23.900547 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.900505    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7" event={"ID":"c11f1e3c-d8be-4fbe-80e3-64496f40683c","Type":"ContainerStarted","Data":"bbd3eaf7be97c379cabb217c89fc0333a88c682161b8449075d4554ca8efec16"}
Nov 02 17:54:23.900764 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.900573    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7" event={"ID":"c11f1e3c-d8be-4fbe-80e3-64496f40683c","Type":"ContainerStarted","Data":"dd1485ca52ee8cffb9fad9b584626bd4b38132482b98e1db987e8410e14b1daa"}
Nov 02 17:54:23.900764 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.900751    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:23.903285 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.903259    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw" event={"ID":"e8ae4f0e-5ec5-4194-9971-5e8d04955e22","Type":"ContainerStarted","Data":"0b2ef9c4242f3924124d4d067f6714f9b4aecef006f64431b487917c9addfcc9"}
Nov 02 17:54:23.903390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:23.903361    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:24.015825 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:24.015711    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7" podStartSLOduration=2.015692706 podStartE2EDuration="2.015692706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:24.014176518 +0000 UTC m=+1894.854782453" watchObservedRunningTime="2026-11-02 17:54:24.015692706 +0000 UTC m=+1894.856298639"
Nov 02 17:54:24.132135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:24.132072    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw" podStartSLOduration=2.1320503889999998 podStartE2EDuration="2.132050389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:24.127978782 +0000 UTC m=+1894.968584713" watchObservedRunningTime="2026-11-02 17:54:24.132050389 +0000 UTC m=+1894.972656324"
Nov 02 17:54:28.497908 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:28.497873    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:29.061186 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:29.061143    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:54:33.306757 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:33.306724    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:34.914181 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:34.914150    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:54:34.914552 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:34.914215    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:54:37.536363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:37.536330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:37.736881 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:37.736851    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:38.202671 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:38.202621    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:41.415533 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:41.415489    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:41.999035 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:41.998999    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"]
Nov 02 17:54:42.431290 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:42.431252    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:42.641118 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:42.641085    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:43.207233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:43.207198    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:45.014236 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:45.014164    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:46.999827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:46.999770    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt" podUID="02e716ae-087a-44d7-8f34-c46ddf92787b" containerName="payload-processing" containerID="cri-o://9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c" gracePeriod=25
Nov 02 17:54:47.249788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.249759    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:54:47.413036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.413008    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk\" (UniqueName: \"kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk\") pod \"02e716ae-087a-44d7-8f34-c46ddf92787b\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") "
Nov 02 17:54:47.413207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.413172    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume\") pod \"02e716ae-087a-44d7-8f34-c46ddf92787b\" (UID: \"02e716ae-087a-44d7-8f34-c46ddf92787b\") "
Nov 02 17:54:47.413659 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.413605    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume" pod "02e716ae-087a-44d7-8f34-c46ddf92787b" (UID: "02e716ae-087a-44d7-8f34-c46ddf92787b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:47.415326 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.415305    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk" pod "02e716ae-087a-44d7-8f34-c46ddf92787b" (UID: "02e716ae-087a-44d7-8f34-c46ddf92787b"). InnerVolumeSpecName "kube-api-access-6zxjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:47.514233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.514198    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxjk\" (UniqueName: \"kubernetes.io/projected/02e716ae-087a-44d7-8f34-c46ddf92787b-kube-api-access-6zxjk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:47.514233 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:47.514223    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02e716ae-087a-44d7-8f34-c46ddf92787b-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:54:48.077435 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.077394    4476 generic.go:356] "Generic (PLEG): container finished" podID="02e716ae-087a-44d7-8f34-c46ddf92787b" containerID="9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c" exitCode=0
Nov 02 17:54:48.077900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.077453    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"
Nov 02 17:54:48.077900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.077480    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt" event={"ID":"02e716ae-087a-44d7-8f34-c46ddf92787b","Type":"ContainerDied","Data":"9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c"}
Nov 02 17:54:48.077900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.077540    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt" event={"ID":"02e716ae-087a-44d7-8f34-c46ddf92787b","Type":"ContainerDied","Data":"209de0a2b0c43f08dea8813850b099932b071d09bb34c1cbb27a7238dc757db6"}
Nov 02 17:54:48.077900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.077564    4476 scope.go:122] "RemoveContainer" containerID="9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c"
Nov 02 17:54:48.088085 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.088068    4476 scope.go:122] "RemoveContainer" containerID="9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c"
Nov 02 17:54:48.088388 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:48.088367    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c\": container with ID starting with 9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c not found: ID does not exist" containerID="9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c"
Nov 02 17:54:48.088434 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.088396    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c"} err="failed to get container status \"9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c\": rpc error: code = NotFound desc = could not find container \"9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c\": container with ID starting with 9ecb74bb3a6e3399421f3477a7904deea2438eea6d173de41d7022120d495a0c not found: ID does not exist"
Nov 02 17:54:48.242149 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.242118    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"]
Nov 02 17:54:48.291271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:48.291237    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-666de0-c7895479c-phsnt"]
Nov 02 17:54:49.172067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:49.169431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:49.390571 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:49.390541    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e716ae-087a-44d7-8f34-c46ddf92787b" path="/var/lib/kubelet/pods/02e716ae-087a-44d7-8f34-c46ddf92787b/volumes"
Nov 02 17:54:50.238572 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:50.238468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:50.695223 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:50.695188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:53.992423 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:53.992379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:54.501356 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:54.501308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:54.713759 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:54.713698    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:55.451591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:55.451561    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:56.134268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:56.134230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:56.369788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:56.369747    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:57.298705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:54:57.298504    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:59.062442 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:54:59.062406    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:55:00.644115 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:00.644003    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:00.891044 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:00.890990    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:01.154390 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:01.154329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:02.443343 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:02.443291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:04.279987 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.279941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:04.480979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.480936    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:04.658232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.658196    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"]
Nov 02 17:55:04.658949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.658921    4476 cpu_manager.go:414] "containerMap: removing container" podUID="02e716ae-087a-44d7-8f34-c46ddf92787b" containerName="payload-processing"
Nov 02 17:55:04.658949 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.658941    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e716ae-087a-44d7-8f34-c46ddf92787b" containerName="payload-processing"
Nov 02 17:55:04.659134 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.659095    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e716ae-087a-44d7-8f34-c46ddf92787b" containerName="payload-processing"
Nov 02 17:55:04.664260 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.664243    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.718219 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.718184    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"]
Nov 02 17:55:04.718437 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.718412    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4z8\" (UniqueName: \"kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8\") pod \"payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.718663 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.718622    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.764259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.764223    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1cea7c\""
Nov 02 17:55:04.764477 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.764377    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1cea7c-dockercfg-gmhg4\""
Nov 02 17:55:04.820379 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.820339    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4z8\" (UniqueName: \"kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8\") pod \"payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.820877 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.820851    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.821332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.821312    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"]
Nov 02 17:55:04.821738 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.821715    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.833707 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.833678    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:04.875201 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.875167    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"]
Nov 02 17:55:04.887523 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.887447    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4z8\" (UniqueName: \"kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8\") pod \"payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:04.922070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.921998    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwwm\" (UniqueName: \"kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm\") pod \"payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:04.922213 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.922088    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:04.979201 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:04.979166    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:05.023207 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.023169    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwwm\" (UniqueName: \"kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm\") pod \"payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:05.023366 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.023238    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:05.024050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.024017    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:05.093693 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.083535    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwwm\" (UniqueName: \"kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm\") pod \"payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:05.167994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.167955    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:05.367920 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.367892    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"]
Nov 02 17:55:05.370089 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:55:05.370062    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd826bdb0_881e_4479_9cb6_361a583919bf.slice/crio-ab536a368bfae721e9140308051a3fb7402b61c29f45eb13a4c1bb6489c55984 WatchSource:0}: Error finding container ab536a368bfae721e9140308051a3fb7402b61c29f45eb13a4c1bb6489c55984: Status 404 returned error can't find the container with id ab536a368bfae721e9140308051a3fb7402b61c29f45eb13a4c1bb6489c55984
Nov 02 17:55:05.548847 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:05.548812    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"]
Nov 02 17:55:05.549116 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:55:05.549093    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dec886_06b7_4f44_b095_ab8018293f93.slice/crio-c519d536a12b3f0e5f569e8cac05f7d5f820d519c264bae359768d26442d3f94 WatchSource:0}: Error finding container c519d536a12b3f0e5f569e8cac05f7d5f820d519c264bae359768d26442d3f94: Status 404 returned error can't find the container with id c519d536a12b3f0e5f569e8cac05f7d5f820d519c264bae359768d26442d3f94
Nov 02 17:55:06.220264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.220230    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd" event={"ID":"a0dec886-06b7-4f44-b095-ab8018293f93","Type":"ContainerStarted","Data":"6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412"}
Nov 02 17:55:06.220460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.220300    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:06.220460 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.220320    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd" event={"ID":"a0dec886-06b7-4f44-b095-ab8018293f93","Type":"ContainerStarted","Data":"c519d536a12b3f0e5f569e8cac05f7d5f820d519c264bae359768d26442d3f94"}
Nov 02 17:55:06.222071 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.222049    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t" event={"ID":"d826bdb0-881e-4479-9cb6-361a583919bf","Type":"ContainerStarted","Data":"faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa"}
Nov 02 17:55:06.222171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.222098    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t" event={"ID":"d826bdb0-881e-4479-9cb6-361a583919bf","Type":"ContainerStarted","Data":"ab536a368bfae721e9140308051a3fb7402b61c29f45eb13a4c1bb6489c55984"}
Nov 02 17:55:06.222171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.222130    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:06.336570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.336519    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd" podStartSLOduration=2.336500876 podStartE2EDuration="2.336500876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:06.332408688 +0000 UTC m=+1937.173014619" watchObservedRunningTime="2026-11-02 17:55:06.336500876 +0000 UTC m=+1937.177106810"
Nov 02 17:55:06.448355 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:06.445907    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t" podStartSLOduration=2.445888177 podStartE2EDuration="2.445888177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:06.444699713 +0000 UTC m=+1937.285305650" watchObservedRunningTime="2026-11-02 17:55:06.445888177 +0000 UTC m=+1937.286494109"
Nov 02 17:55:07.579468 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:07.579434    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:09.012837 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:09.012800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:09.208961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:09.208929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:11.487814 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:11.487773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:12.100657 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:12.091519    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:12.431320 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:12.431268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:12.669092 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:12.669058    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:15.933042 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:15.933004    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:16.217654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:16.217515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:16.413442 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:16.413406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:16.825977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:16.825941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:17.270651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.270327    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:55:17.274067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.274036    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:55:17.880356 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.880324    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"]
Nov 02 17:55:17.884348 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.884327    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:17.936155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.936128    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f10385\""
Nov 02 17:55:17.940580 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.940551    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"]
Nov 02 17:55:17.984792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.984770    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f10385-dockercfg-gm4q7\""
Nov 02 17:55:17.993128 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.993103    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:17.993265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:17.993241    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzn8k\" (UniqueName: \"kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k\") pod \"payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:18.094285 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.094237    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzn8k\" (UniqueName: \"kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k\") pod \"payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:18.094463 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.094379    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:18.095076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.095057    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:18.150532 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.150475    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzn8k\" (UniqueName: \"kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k\") pod \"payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:18.198473 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.198445    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:18.571421 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.571343    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"]
Nov 02 17:55:18.575743 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.575721    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.581714 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.581694    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"]
Nov 02 17:55:18.582887 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:55:18.582861    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d9c50a_9580_4ae1_8773_1b8e6fb745f0.slice/crio-5fb844e60cca1f2bb63054a6a46208fa42dcb46eba4106908b13c7a4a6ef1c01 WatchSource:0}: Error finding container 5fb844e60cca1f2bb63054a6a46208fa42dcb46eba4106908b13c7a4a6ef1c01: Status 404 returned error can't find the container with id 5fb844e60cca1f2bb63054a6a46208fa42dcb46eba4106908b13c7a4a6ef1c01
Nov 02 17:55:18.631678 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.631654    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"]
Nov 02 17:55:18.699913 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.699876    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.700099 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.700071    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgkm\" (UniqueName: \"kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm\") pod \"payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.800930 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.800896    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgkm\" (UniqueName: \"kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm\") pod \"payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.801082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.801000    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.801652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.801616    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.857887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.857824    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgkm\" (UniqueName: \"kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm\") pod \"payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:18.901996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:18.901973    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:19.181644 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.181590    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:19.285649 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.285154    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"]
Nov 02 17:55:19.305152 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:55:19.305109    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ac817f_06ba_4d5d_a6ba_9e099f70b73f.slice/crio-85f735ca9c138b2669796737a9d193fc67c84e343c7c80f0457bf9a747a875e8 WatchSource:0}: Error finding container 85f735ca9c138b2669796737a9d193fc67c84e343c7c80f0457bf9a747a875e8: Status 404 returned error can't find the container with id 85f735ca9c138b2669796737a9d193fc67c84e343c7c80f0457bf9a747a875e8
Nov 02 17:55:19.337912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.337872    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb" event={"ID":"82d9c50a-9580-4ae1-8773-1b8e6fb745f0","Type":"ContainerStarted","Data":"5fb844e60cca1f2bb63054a6a46208fa42dcb46eba4106908b13c7a4a6ef1c01"}
Nov 02 17:55:19.339176 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.339144    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b" event={"ID":"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f","Type":"ContainerStarted","Data":"85f735ca9c138b2669796737a9d193fc67c84e343c7c80f0457bf9a747a875e8"}
Nov 02 17:55:19.389423 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.389390    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:19.572252 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.572177    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"]
Nov 02 17:55:19.577331 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.577311    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.629859 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.629831    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9ac699\""
Nov 02 17:55:19.630036 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.629951    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9ac699-dockercfg-27sj9\""
Nov 02 17:55:19.633135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.633112    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"]
Nov 02 17:55:19.721668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.719865    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j68s6\" (UniqueName: \"kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6\") pod \"payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.721668 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.719965    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.748568 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.748375    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"]
Nov 02 17:55:19.755247 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.755220    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:19.771713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.771682    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"]
Nov 02 17:55:19.812450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.812379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:19.820969 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.820944    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:19.821117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.820991    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68s6\" (UniqueName: \"kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6\") pod \"payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.821117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.821094    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnng\" (UniqueName: \"kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng\") pod \"payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:19.821249 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.821121    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.821970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.821943    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.883318 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.878926    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68s6\" (UniqueName: \"kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6\") pod \"payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.891096 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.891065    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:19.922337 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.922295    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnng\" (UniqueName: \"kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng\") pod \"payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:19.922538 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.922518    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:19.923428 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.923401    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:19.981063 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:19.981037    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnng\" (UniqueName: \"kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng\") pod \"payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:20.074797 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:20.074763    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:20.282315 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:20.282278    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"]
Nov 02 17:55:20.283176 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:55:20.283141    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca8c20f_c6b8_4707_a087_029ac84e4d92.slice/crio-2e8d2974faad7a55e0c193d1312c4e7e64bd9f7c3442b4d9325b62745fe98d1c WatchSource:0}: Error finding container 2e8d2974faad7a55e0c193d1312c4e7e64bd9f7c3442b4d9325b62745fe98d1c: Status 404 returned error can't find the container with id 2e8d2974faad7a55e0c193d1312c4e7e64bd9f7c3442b4d9325b62745fe98d1c
Nov 02 17:55:20.354729 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:20.354687    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw" event={"ID":"fca8c20f-c6b8-4707-a087-029ac84e4d92","Type":"ContainerStarted","Data":"2e8d2974faad7a55e0c193d1312c4e7e64bd9f7c3442b4d9325b62745fe98d1c"}
Nov 02 17:55:20.457430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:20.456943    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"]
Nov 02 17:55:21.365310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.365165    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw" event={"ID":"fca8c20f-c6b8-4707-a087-029ac84e4d92","Type":"ContainerStarted","Data":"9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5"}
Nov 02 17:55:21.365310 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.365271    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:21.368001 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.367976    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf" event={"ID":"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570","Type":"ContainerStarted","Data":"4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6"}
Nov 02 17:55:21.368147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.368026    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf" event={"ID":"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570","Type":"ContainerStarted","Data":"28ee1f47d75e9de2d05d16de573f177f845987cf54786e6bb34c1b475b13dcb9"}
Nov 02 17:55:21.368147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.368073    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:21.369573 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.369544    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb" event={"ID":"82d9c50a-9580-4ae1-8773-1b8e6fb745f0","Type":"ContainerStarted","Data":"bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e"}
Nov 02 17:55:21.369698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.369598    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:21.371308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.371289    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b" event={"ID":"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f","Type":"ContainerStarted","Data":"1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79"}
Nov 02 17:55:21.371405 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.371337    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:21.483692 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.483617    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw" podStartSLOduration=2.483601685 podStartE2EDuration="2.483601685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:21.480110539 +0000 UTC m=+1952.320716538" watchObservedRunningTime="2026-11-02 17:55:21.483601685 +0000 UTC m=+1952.324207619"
Nov 02 17:55:21.734927 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.729144    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf" podStartSLOduration=2.7291224229999997 podStartE2EDuration="2.729122423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:21.594790596 +0000 UTC m=+1952.435396533" watchObservedRunningTime="2026-11-02 17:55:21.729122423 +0000 UTC m=+1952.569728360"
Nov 02 17:55:21.849651 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.847081    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b" podStartSLOduration=2.150039098 podStartE2EDuration="3.847062507s" totalImagesPullingTime="1.697023409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:21.730584846 +0000 UTC m=+1952.571190820" watchObservedRunningTime="2026-11-02 17:55:21.847062507 +0000 UTC m=+1952.687668439"
Nov 02 17:55:21.856149 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:21.854056    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb" podStartSLOduration=2.678319249 podStartE2EDuration="4.854042315s" totalImagesPullingTime="2.175723066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:21.842953591 +0000 UTC m=+1952.683559528" watchObservedRunningTime="2026-11-02 17:55:21.854042315 +0000 UTC m=+1952.694648250"
Nov 02 17:55:24.633451 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:24.633398    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:24.817993 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:24.817955    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:25.006611 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:25.006530    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:28.985645 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:28.985593    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:29.092002 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:55:29.091962    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:55:29.554641 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:29.554590    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:29.753673 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:29.753619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:30.872547 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:30.872499    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:31.089341 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:31.089295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:32.381660 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.381592    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:55:32.382132 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.381706    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:55:32.382735 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.382712    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:55:32.383152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.383131    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:55:32.493033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.492984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:32.921385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.921346    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"]
Nov 02 17:55:32.921803 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.921751    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy" containerID="cri-o://117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104" gracePeriod=30
Nov 02 17:55:32.930996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.930963    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"]
Nov 02 17:55:32.931222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.931196    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy" containerID="cri-o://baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af" gracePeriod=30
Nov 02 17:55:32.993463 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:32.993429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:33.794526 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:33.794486    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:35.905401 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:35.905367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:36.105364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:36.105333    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:38.213200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.213169    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:55:38.253884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.253858    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:55:38.267447 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267418    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267454    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267475    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267506    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.267595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267551    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267582    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267677    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267703    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267739    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267766    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267772    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267801    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267854    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267869    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.267914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267913    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.267999    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268036    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268042    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268069    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9\") pod \"ecda5590-9500-4f46-a220-894b04a9bc47\" (UID: \"ecda5590-9500-4f46-a220-894b04a9bc47\") "
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268162    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268198    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data\") pod \"e7069351-3bef-481e-a809-acc2f2b27c83\" (UID: \"e7069351-3bef-481e-a809-acc2f2b27c83\") "
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268252    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268271    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.268385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268297    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:38.268887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268752    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.268887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268830    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:38.268989 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268927    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.269051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.268996    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269016    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269032    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269051 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269046    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269061    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7069351-3bef-481e-a809-acc2f2b27c83-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269075    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269090    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269257 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269104    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecda5590-9500-4f46-a220-894b04a9bc47-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.269461 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.269306    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.270755 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.270690    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.270875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.270848    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.271282 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.271252    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "kube-api-access-n27vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.272314 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.272284    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:38.272490 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.272466    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token" pod "e7069351-3bef-481e-a809-acc2f2b27c83" (UID: "e7069351-3bef-481e-a809-acc2f2b27c83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.272534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.272500    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "kube-api-access-tkhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:38.273196 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.273180    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:38.273287 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.273272    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo" pod "ecda5590-9500-4f46-a220-894b04a9bc47" (UID: "ecda5590-9500-4f46-a220-894b04a9bc47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:38.369925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369891    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.369925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369919    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.369925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369930    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecda5590-9500-4f46-a220-894b04a9bc47-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369939    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27vc\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-kube-api-access-n27vc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369949    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhh9\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-kube-api-access-tkhh9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369958    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7069351-3bef-481e-a809-acc2f2b27c83-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369966    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7069351-3bef-481e-a809-acc2f2b27c83-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369975    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecda5590-9500-4f46-a220-894b04a9bc47-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369984    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecda5590-9500-4f46-a220-894b04a9bc47-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.370154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.369993    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7069351-3bef-481e-a809-acc2f2b27c83-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.514253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.514162    4476 generic.go:356] "Generic (PLEG): container finished" podID="e7069351-3bef-481e-a809-acc2f2b27c83" containerID="117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104" exitCode=0
Nov 02 17:55:38.514253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.514211    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"
Nov 02 17:55:38.514253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.514215    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" event={"ID":"e7069351-3bef-481e-a809-acc2f2b27c83","Type":"ContainerDied","Data":"117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104"}
Nov 02 17:55:38.514537 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.514280    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw" event={"ID":"e7069351-3bef-481e-a809-acc2f2b27c83","Type":"ContainerDied","Data":"5e168cf0bdb0923d1484a675dace1dc71867708cc490f0a4f863c975e075a79b"}
Nov 02 17:55:38.514537 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.514311    4476 scope.go:122] "RemoveContainer" containerID="117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104"
Nov 02 17:55:38.525160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.516372    4476 generic.go:356] "Generic (PLEG): container finished" podID="ecda5590-9500-4f46-a220-894b04a9bc47" containerID="baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af" exitCode=0
Nov 02 17:55:38.525160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.516418    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" event={"ID":"ecda5590-9500-4f46-a220-894b04a9bc47","Type":"ContainerDied","Data":"baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af"}
Nov 02 17:55:38.525160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.516462    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn" event={"ID":"ecda5590-9500-4f46-a220-894b04a9bc47","Type":"ContainerDied","Data":"c4dae8aa1918c373d63ecdfa23f89f5a4f3af56c9e2940567eae9ae87e2c2eea"}
Nov 02 17:55:38.525160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.516471    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"
Nov 02 17:55:38.528027 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.527856    4476 scope.go:122] "RemoveContainer" containerID="117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104"
Nov 02 17:55:38.528177 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:55:38.528158    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104\": container with ID starting with 117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104 not found: ID does not exist" containerID="117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104"
Nov 02 17:55:38.528254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.528191    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104"} err="failed to get container status \"117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104\": rpc error: code = NotFound desc = could not find container \"117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104\": container with ID starting with 117146419a365871fbe29818164edcb51f3d9af4dc5419c7d241c7592c090104 not found: ID does not exist"
Nov 02 17:55:38.528254 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.528215    4476 scope.go:122] "RemoveContainer" containerID="baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af"
Nov 02 17:55:38.538572 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.538551    4476 scope.go:122] "RemoveContainer" containerID="baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af"
Nov 02 17:55:38.538882 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:55:38.538858    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af\": container with ID starting with baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af not found: ID does not exist" containerID="baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af"
Nov 02 17:55:38.538964 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.538895    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af"} err="failed to get container status \"baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af\": rpc error: code = NotFound desc = could not find container \"baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af\": container with ID starting with baae097879ee0c0fb0cec096e095ba0dd7b742d5ae596a6aef191f240ee725af not found: ID does not exist"
Nov 02 17:55:38.680919 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.680889    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"]
Nov 02 17:55:38.684853 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.684822    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-cs9jw"]
Nov 02 17:55:38.798301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.798220    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"]
Nov 02 17:55:38.802875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:38.802837    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-w8dmn"]
Nov 02 17:55:39.383956 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:39.383916    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" path="/var/lib/kubelet/pods/e7069351-3bef-481e-a809-acc2f2b27c83/volumes"
Nov 02 17:55:39.384361 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:39.384346    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" path="/var/lib/kubelet/pods/ecda5590-9500-4f46-a220-894b04a9bc47/volumes"
Nov 02 17:55:39.808554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:39.808473    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:40.647942 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:40.647899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:41.846061 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:41.846024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:42.163805 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:42.163770    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:42.445715 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:42.437865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:42.633099 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:42.633069    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:48.903040 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:48.903011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:49.090026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:49.089998    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:49.296007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:49.295919    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:49.583958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:49.583847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:49.830971 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:49.830932    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:50.080622 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:50.080582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:50.263844 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:50.263747    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:50.794742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:50.794709    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:50.974164 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:50.974130    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:51.885398 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:51.885356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:54.381139 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:54.381103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:54.602372 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:54.602341    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:56.087093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:56.087053    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:57.045187 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:57.045152    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:57.626958 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:57.626928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:57.821294 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:55:57.821264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:59.059828 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:55:59.059791    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:56:01.157591 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:01.157547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:01.373696 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:01.373617    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:01.850408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:01.849289    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": read tcp 10.133.0.2:40872->10.133.0.115:15021: read: connection reset by peer" start-of-body=
Nov 02 17:56:01.850408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:01.849370    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": read tcp 10.133.0.2:40872->10.133.0.115:15021: read: connection reset by peer"
Nov 02 17:56:02.732215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:02.732181    4476 generic.go:356] "Generic (PLEG): container finished" podID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerID="2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2" exitCode=0
Nov 02 17:56:02.732621 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:02.732257    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" event={"ID":"ab0c4a68-ea6a-43ea-958a-87be9d86be8f","Type":"ContainerDied","Data":"2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"}
Nov 02 17:56:02.732761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:02.732746    4476 scope.go:122] "RemoveContainer" containerID="2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"
Nov 02 17:56:02.734264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:02.734240    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:56:02.734332 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:02.734318    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:56:02.734368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:02.734355    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:56:03.757407 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:03.757369    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" event={"ID":"ab0c4a68-ea6a-43ea-958a-87be9d86be8f","Type":"ContainerStarted","Data":"0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd"}
Nov 02 17:56:03.874371 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:03.874342    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:56:03.874540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:03.874492    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:56:03.874540 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:03.874528    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 17:56:04.874267 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:04.874235    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:56:04.874766 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:04.874323    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 17:56:05.874429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:05.874391    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:56:05.874940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:05.874454    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 17:56:06.493561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:06.493530    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:06.874304 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:06.874271    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:56:06.874482 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:06.874327    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 17:56:07.874937 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:07.874902    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:56:07.875350 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:07.874958    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 17:56:08.874308 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:08.874270    4476 patch_prober.go:29] interesting pod/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:56:08.874527 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:08.874338    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 02 17:56:09.878915 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:09.878888    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:56:09.879329 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:09.879318    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:56:09.879923 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:09.879907    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 17:56:11.482375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:11.482325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:15.746278 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:15.746240    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:16.066253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.066172    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"]
Nov 02 17:56:16.066417 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.066383    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b" podUID="e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" containerName="payload-processing" containerID="cri-o://1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79" gracePeriod=30
Nov 02 17:56:16.121872 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.121842    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"]
Nov 02 17:56:16.122050 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.122029    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb" podUID="82d9c50a-9580-4ae1-8773-1b8e6fb745f0" containerName="payload-pre-processing" containerID="cri-o://bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e" gracePeriod=30
Nov 02 17:56:16.272075 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.272039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:16.404864 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.404838    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:56:16.432503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.432479    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:56:16.453076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.453042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:16.480226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.480201    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k\" (UniqueName: \"kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k\") pod \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") "
Nov 02 17:56:16.480352 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.480249    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume\") pod \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") "
Nov 02 17:56:16.480545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.480522    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume" pod "e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" (UID: "e8ac817f-06ba-4d5d-a6ba-9e099f70b73f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:16.480683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.480557    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume\") pod \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\" (UID: \"82d9c50a-9580-4ae1-8773-1b8e6fb745f0\") "
Nov 02 17:56:16.480789 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.480760    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm\" (UniqueName: \"kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm\") pod \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\" (UID: \"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f\") "
Nov 02 17:56:16.481026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.481004    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume" pod "82d9c50a-9580-4ae1-8773-1b8e6fb745f0" (UID: "82d9c50a-9580-4ae1-8773-1b8e6fb745f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:16.481347 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.481321    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:16.481439 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.481352    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:16.483227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.483206    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k" pod "82d9c50a-9580-4ae1-8773-1b8e6fb745f0" (UID: "82d9c50a-9580-4ae1-8773-1b8e6fb745f0"). InnerVolumeSpecName "kube-api-access-wzn8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:16.483439 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.483425    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm" pod "e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" (UID: "e8ac817f-06ba-4d5d-a6ba-9e099f70b73f"). InnerVolumeSpecName "kube-api-access-kkgkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:16.583297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.583180    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkgkm\" (UniqueName: \"kubernetes.io/projected/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f-kube-api-access-kkgkm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:16.583297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.583217    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzn8k\" (UniqueName: \"kubernetes.io/projected/82d9c50a-9580-4ae1-8773-1b8e6fb745f0-kube-api-access-wzn8k\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:16.855497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.855408    4476 generic.go:356] "Generic (PLEG): container finished" podID="82d9c50a-9580-4ae1-8773-1b8e6fb745f0" containerID="bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e" exitCode=0
Nov 02 17:56:16.855497 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.855470    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"
Nov 02 17:56:16.856093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.855496    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb" event={"ID":"82d9c50a-9580-4ae1-8773-1b8e6fb745f0","Type":"ContainerDied","Data":"bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e"}
Nov 02 17:56:16.856093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.855556    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb" event={"ID":"82d9c50a-9580-4ae1-8773-1b8e6fb745f0","Type":"ContainerDied","Data":"5fb844e60cca1f2bb63054a6a46208fa42dcb46eba4106908b13c7a4a6ef1c01"}
Nov 02 17:56:16.856093 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.855590    4476 scope.go:122] "RemoveContainer" containerID="bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e"
Nov 02 17:56:16.857143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.857103    4476 generic.go:356] "Generic (PLEG): container finished" podID="e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" containerID="1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79" exitCode=0
Nov 02 17:56:16.857250 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.857179    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b" event={"ID":"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f","Type":"ContainerDied","Data":"1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79"}
Nov 02 17:56:16.857250 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.857227    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b" event={"ID":"e8ac817f-06ba-4d5d-a6ba-9e099f70b73f","Type":"ContainerDied","Data":"85f735ca9c138b2669796737a9d193fc67c84e343c7c80f0457bf9a747a875e8"}
Nov 02 17:56:16.857344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.857272    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"
Nov 02 17:56:16.868553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.868532    4476 scope.go:122] "RemoveContainer" containerID="bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e"
Nov 02 17:56:16.868881 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:56:16.868855    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e\": container with ID starting with bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e not found: ID does not exist" containerID="bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e"
Nov 02 17:56:16.869070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.869031    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e"} err="failed to get container status \"bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e\": rpc error: code = NotFound desc = could not find container \"bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e\": container with ID starting with bf963cdd18d7d43e4b3fa8717cf5d1827601a6e692314bf6dcafacecd99b3c3e not found: ID does not exist"
Nov 02 17:56:16.869070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.869064    4476 scope.go:122] "RemoveContainer" containerID="1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79"
Nov 02 17:56:16.881592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.881574    4476 scope.go:122] "RemoveContainer" containerID="1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79"
Nov 02 17:56:16.881931 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:56:16.881913    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79\": container with ID starting with 1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79 not found: ID does not exist" containerID="1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79"
Nov 02 17:56:16.881994 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:16.881938    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79"} err="failed to get container status \"1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79\": rpc error: code = NotFound desc = could not find container \"1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79\": container with ID starting with 1495431c4772420d5945fb9fa635d207f516d63873ded8e7bf04e68c1b1a7e79 not found: ID does not exist"
Nov 02 17:56:17.019112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.019044    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"]
Nov 02 17:56:17.023305 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.023279    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f10385-66df4f4c9d-v5h6b"]
Nov 02 17:56:17.179788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.179758    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"]
Nov 02 17:56:17.189791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.189765    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f10385-74b88c7d87-lqgcb"]
Nov 02 17:56:17.386086 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.386041    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d9c50a-9580-4ae1-8773-1b8e6fb745f0" path="/var/lib/kubelet/pods/82d9c50a-9580-4ae1-8773-1b8e6fb745f0/volumes"
Nov 02 17:56:17.386568 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.386551    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" path="/var/lib/kubelet/pods/e8ac817f-06ba-4d5d-a6ba-9e099f70b73f/volumes"
Nov 02 17:56:17.431983 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.431902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:17.653281 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:17.653250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:20.620683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:20.615658    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:20.816082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:20.816047    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:22.189171 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:22.189139    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:22.673996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:22.673964    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:22.920961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:22.920926    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:24.098327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:24.098294    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:24.699967 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:24.699858    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:25.101174 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:25.101073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:27.964701 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:27.964663    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:28.853857 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:28.853820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:29.050448 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:29.050420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:29.060297 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:56:29.060254    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:56:30.154811 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:30.154774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:30.341363 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:30.341322    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:32.047891 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:32.047850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:32.236200 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:32.236167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:33.479267 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:33.479229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:35.875024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:35.874980    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:36.760887 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:36.760854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:37.712000 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:37.691172    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:37.782227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:37.782190    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"]
Nov 02 17:56:37.891227 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:37.891196    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"]
Nov 02 17:56:38.049323 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:38.049236    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:38.264594 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:38.264558    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:38.472506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:38.472469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:38.679265 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:38.679233    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:42.783084 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:42.783038    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7" podUID="c11f1e3c-d8be-4fbe-80e3-64496f40683c" containerName="payload-processing" containerID="cri-o://bbd3eaf7be97c379cabb217c89fc0333a88c682161b8449075d4554ca8efec16" gracePeriod=25
Nov 02 17:56:42.891709 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:42.891657    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw" podUID="e8ae4f0e-5ec5-4194-9971-5e8d04955e22" containerName="payload-pre-processing" containerID="cri-o://0b2ef9c4242f3924124d4d067f6714f9b4aecef006f64431b487917c9addfcc9" gracePeriod=25
Nov 02 17:56:43.067389 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.067349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:43.069226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.069203    4476 generic.go:356] "Generic (PLEG): container finished" podID="c11f1e3c-d8be-4fbe-80e3-64496f40683c" containerID="bbd3eaf7be97c379cabb217c89fc0333a88c682161b8449075d4554ca8efec16" exitCode=0
Nov 02 17:56:43.069344 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.069303    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7" event={"ID":"c11f1e3c-d8be-4fbe-80e3-64496f40683c","Type":"ContainerDied","Data":"bbd3eaf7be97c379cabb217c89fc0333a88c682161b8449075d4554ca8efec16"}
Nov 02 17:56:43.069429 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.069397    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7" event={"ID":"c11f1e3c-d8be-4fbe-80e3-64496f40683c","Type":"ContainerDied","Data":"dd1485ca52ee8cffb9fad9b584626bd4b38132482b98e1db987e8410e14b1daa"}
Nov 02 17:56:43.069558 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.069441    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1485ca52ee8cffb9fad9b584626bd4b38132482b98e1db987e8410e14b1daa"
Nov 02 17:56:43.071652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.070995    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:56:43.071652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.071078    4476 generic.go:356] "Generic (PLEG): container finished" podID="e8ae4f0e-5ec5-4194-9971-5e8d04955e22" containerID="0b2ef9c4242f3924124d4d067f6714f9b4aecef006f64431b487917c9addfcc9" exitCode=0
Nov 02 17:56:43.071652 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.071274    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw" event={"ID":"e8ae4f0e-5ec5-4194-9971-5e8d04955e22","Type":"ContainerDied","Data":"0b2ef9c4242f3924124d4d067f6714f9b4aecef006f64431b487917c9addfcc9"}
Nov 02 17:56:43.114376 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.114336    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp\" (UniqueName: \"kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp\") pod \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") "
Nov 02 17:56:43.114546 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.114440    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume\") pod \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\" (UID: \"c11f1e3c-d8be-4fbe-80e3-64496f40683c\") "
Nov 02 17:56:43.115082 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.115057    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume" pod "c11f1e3c-d8be-4fbe-80e3-64496f40683c" (UID: "c11f1e3c-d8be-4fbe-80e3-64496f40683c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:43.117840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.117805    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp" pod "c11f1e3c-d8be-4fbe-80e3-64496f40683c" (UID: "c11f1e3c-d8be-4fbe-80e3-64496f40683c"). InnerVolumeSpecName "kube-api-access-7tmhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:43.182240 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.182213    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:56:43.216467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.216426    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tmhp\" (UniqueName: \"kubernetes.io/projected/c11f1e3c-d8be-4fbe-80e3-64496f40683c-kube-api-access-7tmhp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:43.216467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.216459    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c11f1e3c-d8be-4fbe-80e3-64496f40683c-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:43.328602 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.328495    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk\" (UniqueName: \"kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk\") pod \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") "
Nov 02 17:56:43.328792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.328675    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume\") pod \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\" (UID: \"e8ae4f0e-5ec5-4194-9971-5e8d04955e22\") "
Nov 02 17:56:43.329773 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.329742    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume" pod "e8ae4f0e-5ec5-4194-9971-5e8d04955e22" (UID: "e8ae4f0e-5ec5-4194-9971-5e8d04955e22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:43.331854 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.331819    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk" pod "e8ae4f0e-5ec5-4194-9971-5e8d04955e22" (UID: "e8ae4f0e-5ec5-4194-9971-5e8d04955e22"). InnerVolumeSpecName "kube-api-access-kdgzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:43.388748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.388706    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:43.430551 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.430524    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdgzk\" (UniqueName: \"kubernetes.io/projected/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-kube-api-access-kdgzk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:43.430551 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.430550    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ae4f0e-5ec5-4194-9971-5e8d04955e22-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:56:43.689061 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.689023    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:43.868965 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:43.868927    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:44.081880 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.081796    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"
Nov 02 17:56:44.082061 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.081795    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw" event={"ID":"e8ae4f0e-5ec5-4194-9971-5e8d04955e22","Type":"ContainerDied","Data":"3d866966356970da3e74660e3ce50ae59f5be06f063baf31b91a42c50c298889"}
Nov 02 17:56:44.082061 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.081934    4476 scope.go:122] "RemoveContainer" containerID="0b2ef9c4242f3924124d4d067f6714f9b4aecef006f64431b487917c9addfcc9"
Nov 02 17:56:44.082202 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.082064    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"
Nov 02 17:56:44.247683 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.247648    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"]
Nov 02 17:56:44.297327 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.297282    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-55b9679dd7-tz6sw"]
Nov 02 17:56:44.403595 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.403557    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"]
Nov 02 17:56:44.411038 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:44.411007    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-857bb8989-qqbc7"]
Nov 02 17:56:45.384459 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:45.384424    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11f1e3c-d8be-4fbe-80e3-64496f40683c" path="/var/lib/kubelet/pods/c11f1e3c-d8be-4fbe-80e3-64496f40683c/volumes"
Nov 02 17:56:45.384868 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:45.384853    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ae4f0e-5ec5-4194-9971-5e8d04955e22" path="/var/lib/kubelet/pods/e8ae4f0e-5ec5-4194-9971-5e8d04955e22/volumes"
Nov 02 17:56:46.607708 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:46.607670    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:46.795296 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:46.795258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:49.401876 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:49.401846    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:51.315117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:51.315087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:56.369570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:56.369531    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:56.566335 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:56.566302    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:56.754744 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:56:56.754621    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:59.063101 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:56:59.063062    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:57:01.542467 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:01.542427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:06.389124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:06.389087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:11.194088 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:11.194028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:13.519053 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:13.519011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:16.115070 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:16.115039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:16.312935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:16.312902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:20.882875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:20.882839    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:21.916565 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:21.916513    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:22.132821 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:22.132780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:22.328297 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:22.328213    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:26.446956 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:26.446916    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:28.113197 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:28.113155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:29.060621 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:29.060578    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:57:29.083007 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:29.082967    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:30.524106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:30.524071    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"]
Nov 02 17:57:30.629521 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:30.629492    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"]
Nov 02 17:57:30.953329 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:30.953268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:31.137336 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:31.137305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:31.788333 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:31.788282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:35.525059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.524995    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf" podUID="0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" containerName="payload-processing" containerID="cri-o://4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6" gracePeriod=25
Nov 02 17:57:35.630112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.630048    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw" podUID="fca8c20f-c6b8-4707-a087-029ac84e4d92" containerName="payload-pre-processing" containerID="cri-o://9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5" gracePeriod=25
Nov 02 17:57:35.741068 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.737565    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:35.847931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.847476    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:57:35.974954 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.967131    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng\" (UniqueName: \"kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng\") pod \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") "
Nov 02 17:57:35.974954 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.967335    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume\") pod \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\" (UID: \"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570\") "
Nov 02 17:57:35.998818 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:35.998778    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng" pod "0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" (UID: "0ff07ee1-7e2b-4f51-8e33-9be93f8f7570"). InnerVolumeSpecName "kube-api-access-8cnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:36.003178 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.003141    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume" pod "0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" (UID: "0ff07ee1-7e2b-4f51-8e33-9be93f8f7570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:36.016861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.016835    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:57:36.068301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.068214    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume\") pod \"fca8c20f-c6b8-4707-a087-029ac84e4d92\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") "
Nov 02 17:57:36.068301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.068260    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6\" (UniqueName: \"kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6\") pod \"fca8c20f-c6b8-4707-a087-029ac84e4d92\" (UID: \"fca8c20f-c6b8-4707-a087-029ac84e4d92\") "
Nov 02 17:57:36.068609 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.068588    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume" pod "fca8c20f-c6b8-4707-a087-029ac84e4d92" (UID: "fca8c20f-c6b8-4707-a087-029ac84e4d92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:36.068732 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.068717    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:36.068781 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.068735    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca8c20f-c6b8-4707-a087-029ac84e4d92-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:36.068781 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.068745    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnng\" (UniqueName: \"kubernetes.io/projected/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570-kube-api-access-8cnng\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:36.070726 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.070699    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6" pod "fca8c20f-c6b8-4707-a087-029ac84e4d92" (UID: "fca8c20f-c6b8-4707-a087-029ac84e4d92"). InnerVolumeSpecName "kube-api-access-j68s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:36.170099 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.170054    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j68s6\" (UniqueName: \"kubernetes.io/projected/fca8c20f-c6b8-4707-a087-029ac84e4d92-kube-api-access-j68s6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:36.287728 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.287700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:36.444554 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.444515    4476 generic.go:356] "Generic (PLEG): container finished" podID="fca8c20f-c6b8-4707-a087-029ac84e4d92" containerID="9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5" exitCode=0
Nov 02 17:57:36.444795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.444571    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"
Nov 02 17:57:36.444795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.444617    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw" event={"ID":"fca8c20f-c6b8-4707-a087-029ac84e4d92","Type":"ContainerDied","Data":"9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5"}
Nov 02 17:57:36.444795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.444708    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw" event={"ID":"fca8c20f-c6b8-4707-a087-029ac84e4d92","Type":"ContainerDied","Data":"2e8d2974faad7a55e0c193d1312c4e7e64bd9f7c3442b4d9325b62745fe98d1c"}
Nov 02 17:57:36.444795 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.444740    4476 scope.go:122] "RemoveContainer" containerID="9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5"
Nov 02 17:57:36.446152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.446128    4476 generic.go:356] "Generic (PLEG): container finished" podID="0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" containerID="4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6" exitCode=0
Nov 02 17:57:36.446280 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.446177    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"
Nov 02 17:57:36.446280 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.446197    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf" event={"ID":"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570","Type":"ContainerDied","Data":"4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6"}
Nov 02 17:57:36.446280 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.446241    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf" event={"ID":"0ff07ee1-7e2b-4f51-8e33-9be93f8f7570","Type":"ContainerDied","Data":"28ee1f47d75e9de2d05d16de573f177f845987cf54786e6bb34c1b475b13dcb9"}
Nov 02 17:57:36.457186 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.457170    4476 scope.go:122] "RemoveContainer" containerID="9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5"
Nov 02 17:57:36.457493 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:36.457476    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5\": container with ID starting with 9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5 not found: ID does not exist" containerID="9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5"
Nov 02 17:57:36.457557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.457500    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5"} err="failed to get container status \"9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5\": rpc error: code = NotFound desc = could not find container \"9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5\": container with ID starting with 9813a0398ca5cca3a01ed482b93c874d469d7b22594db12dbd80bc86837e03e5 not found: ID does not exist"
Nov 02 17:57:36.457557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.457517    4476 scope.go:122] "RemoveContainer" containerID="4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6"
Nov 02 17:57:36.468313 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.468291    4476 scope.go:122] "RemoveContainer" containerID="4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6"
Nov 02 17:57:36.468572 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:36.468552    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6\": container with ID starting with 4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6 not found: ID does not exist" containerID="4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6"
Nov 02 17:57:36.468657 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.468583    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6"} err="failed to get container status \"4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6\": rpc error: code = NotFound desc = could not find container \"4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6\": container with ID starting with 4a2ad69dd7c24a7abd48ff91a2bad3474d08fd6457d715e2c2197d1464f23ce6 not found: ID does not exist"
Nov 02 17:57:36.609914 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.609870    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"]
Nov 02 17:57:36.613078 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.613055    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9ac699-b6875cfd5-4fxvf"]
Nov 02 17:57:36.722791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.722704    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"]
Nov 02 17:57:36.725925 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:36.725887    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9ac699-6f67785f79-cfkmw"]
Nov 02 17:57:37.330912 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.330882    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:37.358003 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.357966    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"]
Nov 02 17:57:37.358487 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358474    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fca8c20f-c6b8-4707-a087-029ac84e4d92" containerName="payload-pre-processing"
Nov 02 17:57:37.358534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358489    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca8c20f-c6b8-4707-a087-029ac84e4d92" containerName="payload-pre-processing"
Nov 02 17:57:37.358534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358499    4476 cpu_manager.go:414] "containerMap: removing container" podUID="82d9c50a-9580-4ae1-8773-1b8e6fb745f0" containerName="payload-pre-processing"
Nov 02 17:57:37.358534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358504    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d9c50a-9580-4ae1-8773-1b8e6fb745f0" containerName="payload-pre-processing"
Nov 02 17:57:37.358534 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358533    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" containerName="payload-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358539    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" containerName="payload-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358550    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358555    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358572    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358577    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358586    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c11f1e3c-d8be-4fbe-80e3-64496f40683c" containerName="payload-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358591    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11f1e3c-d8be-4fbe-80e3-64496f40683c" containerName="payload-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358598    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" containerName="payload-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358603    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" containerName="payload-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358611    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e8ae4f0e-5ec5-4194-9971-5e8d04955e22" containerName="payload-pre-processing"
Nov 02 17:57:37.358699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358616    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ae4f0e-5ec5-4194-9971-5e8d04955e22" containerName="payload-pre-processing"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358712    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca8c20f-c6b8-4707-a087-029ac84e4d92" containerName="payload-pre-processing"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358721    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d9c50a-9580-4ae1-8773-1b8e6fb745f0" containerName="payload-pre-processing"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358728    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ac817f-06ba-4d5d-a6ba-9e099f70b73f" containerName="payload-processing"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358733    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecda5590-9500-4f46-a220-894b04a9bc47" containerName="istio-proxy"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358742    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7069351-3bef-481e-a809-acc2f2b27c83" containerName="istio-proxy"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358749    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11f1e3c-d8be-4fbe-80e3-64496f40683c" containerName="payload-processing"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358754    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" containerName="payload-processing"
Nov 02 17:57:37.359073 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.358760    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ae4f0e-5ec5-4194-9971-5e8d04955e22" containerName="payload-pre-processing"
Nov 02 17:57:37.363419 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.363396    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.384503 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.384472    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff07ee1-7e2b-4f51-8e33-9be93f8f7570" path="/var/lib/kubelet/pods/0ff07ee1-7e2b-4f51-8e33-9be93f8f7570/volumes"
Nov 02 17:57:37.385033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.385015    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca8c20f-c6b8-4707-a087-029ac84e4d92" path="/var/lib/kubelet/pods/fca8c20f-c6b8-4707-a087-029ac84e4d92/volumes"
Nov 02 17:57:37.417933 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.417909    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ccf013\""
Nov 02 17:57:37.467152 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.467123    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ccf013-dockercfg-lt8kx\""
Nov 02 17:57:37.470755 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.470728    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"]
Nov 02 17:57:37.483712 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.483679    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwr2z\" (UniqueName: \"kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.483861 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.483739    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.584812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.584736    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwr2z\" (UniqueName: \"kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.584812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.584786    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.585558 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.585528    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.641379 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.641348    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwr2z\" (UniqueName: \"kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z\") pod \"payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:37.677669 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:37.677615    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:38.056375 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.056348    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"]
Nov 02 17:57:38.057709 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:57:38.057682    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daa75d4_fa02_48e1_8981_e9704a3552ed.slice/crio-ec7cbde76c35c9728f96f1a586e452e3a97f891057a7c3d2b3ba82a136257821 WatchSource:0}: Error finding container ec7cbde76c35c9728f96f1a586e452e3a97f891057a7c3d2b3ba82a136257821: Status 404 returned error can't find the container with id ec7cbde76c35c9728f96f1a586e452e3a97f891057a7c3d2b3ba82a136257821
Nov 02 17:57:38.071101 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.071068    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"]
Nov 02 17:57:38.077800 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.077764    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.131833 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.131798    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"]
Nov 02 17:57:38.191970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.191927    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79fm\" (UniqueName: \"kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm\") pod \"payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.192156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.192113    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.293802 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.293759    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.293976 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.293914    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79fm\" (UniqueName: \"kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm\") pod \"payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.294472 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.294452    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.351080 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.351011    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79fm\" (UniqueName: \"kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm\") pod \"payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.422988 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.422955    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:38.467682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.466587    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8" event={"ID":"2daa75d4-fa02-48e1-8981-e9704a3552ed","Type":"ContainerStarted","Data":"fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff"}
Nov 02 17:57:38.467682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.466659    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8" event={"ID":"2daa75d4-fa02-48e1-8981-e9704a3552ed","Type":"ContainerStarted","Data":"ec7cbde76c35c9728f96f1a586e452e3a97f891057a7c3d2b3ba82a136257821"}
Nov 02 17:57:38.467682 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.467488    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:38.580984 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.580919    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8" podStartSLOduration=1.580903495 podStartE2EDuration="1.580903495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:38.580088672 +0000 UTC m=+2089.420694607" watchObservedRunningTime="2026-11-02 17:57:38.580903495 +0000 UTC m=+2089.421509429"
Nov 02 17:57:38.805716 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:38.805687    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"]
Nov 02 17:57:38.807291 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:57:38.807253    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6d359a_e2b8_49d6_baa3_a4fe7b23e8ed.slice/crio-9793aaa1583ec6f8b4816985defea0e3b78cbc4d0994dc04056730ecd4d1b5ab WatchSource:0}: Error finding container 9793aaa1583ec6f8b4816985defea0e3b78cbc4d0994dc04056730ecd4d1b5ab: Status 404 returned error can't find the container with id 9793aaa1583ec6f8b4816985defea0e3b78cbc4d0994dc04056730ecd4d1b5ab
Nov 02 17:57:39.216024 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:39.215984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:39.403676 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:39.403592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:39.479706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:39.476693    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql" event={"ID":"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed","Type":"ContainerStarted","Data":"c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377"}
Nov 02 17:57:39.479706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:39.476753    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql" event={"ID":"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed","Type":"ContainerStarted","Data":"9793aaa1583ec6f8b4816985defea0e3b78cbc4d0994dc04056730ecd4d1b5ab"}
Nov 02 17:57:39.479706 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:39.476979    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:39.592827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:39.592762    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql" podStartSLOduration=1.5927436940000002 podStartE2EDuration="1.592743694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:39.589892722 +0000 UTC m=+2090.430498657" watchObservedRunningTime="2026-11-02 17:57:39.592743694 +0000 UTC m=+2090.433349626"
Nov 02 17:57:44.182253 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:44.182206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:48.966620 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:48.966562    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:49.696713 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:49.696678    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:57:49.705852 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:49.705823    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 17:57:50.487368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:50.487333    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 17:57:50.487840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:50.487421    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 17:57:51.655640 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:51.655583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:52.278026 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:52.277987    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"]
Nov 02 17:57:52.278286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:52.278259    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" containerID="cri-o://c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6" gracePeriod=30
Nov 02 17:57:53.459135 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.459100    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"]
Nov 02 17:57:53.464530 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.464511    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.566979 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.566944    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d5924c-serving-cert\""
Nov 02 17:57:53.570431 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.570407    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"]
Nov 02 17:57:53.576517 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.576483    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.576597 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.576529    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8r4s\" (UniqueName: \"kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.576656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.576615    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.576778 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.576761    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.677767 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.677733    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.677959 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.677776    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8r4s\" (UniqueName: \"kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.677959 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:53.677885    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-serving-cert: secret "maas-api-e2e-ipp-b-d5924c-serving-cert" not found
Nov 02 17:57:53.677959 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:53.677949    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls podName:d5da7c40-e4c1-44cf-b471-8646c8901130 nodeName:}" failed. No retries permitted until 2026-11-02 17:57:54.17792942 +0000 UTC m=+2105.018535332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls") pod "maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" (UID: "d5da7c40-e4c1-44cf-b471-8646c8901130") : secret "maas-api-e2e-ipp-b-d5924c-serving-cert" not found
Nov 02 17:57:53.678158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.677970    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.678158 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.678101    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.678846 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.678775    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.680757 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.680734    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.701228 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.701204    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:53.735656 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.735575    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8r4s\" (UniqueName: \"kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:53.895471 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:53.895433    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:54.176574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:54.176542    4476 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:57:54.176764 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:54.176598    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:57:54.184264 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:54.184236    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:54.186961 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:54.186936    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:54.378261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:54.378229    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:54.770098 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:54.770070    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"]
Nov 02 17:57:54.770657 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:57:54.770614    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5da7c40_e4c1_44cf_b471_8646c8901130.slice/crio-d119730f446894d0571619c42d5790b17b13920e701cc1a50de8d717708e3e9b WatchSource:0}: Error finding container d119730f446894d0571619c42d5790b17b13920e701cc1a50de8d717708e3e9b: Status 404 returned error can't find the container with id d119730f446894d0571619c42d5790b17b13920e701cc1a50de8d717708e3e9b
Nov 02 17:57:55.606049 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:55.606005    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" event={"ID":"d5da7c40-e4c1-44cf-b471-8646c8901130","Type":"ContainerStarted","Data":"d119730f446894d0571619c42d5790b17b13920e701cc1a50de8d717708e3e9b"}
Nov 02 17:57:56.616124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:56.616082    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" event={"ID":"d5da7c40-e4c1-44cf-b471-8646c8901130","Type":"ContainerStarted","Data":"b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d"}
Nov 02 17:57:56.616581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:56.616293    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:57:56.734788 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:56.734704    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" podStartSLOduration=2.896768878 podStartE2EDuration="3.734690386s" totalImagesPullingTime="837.921508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:56.731292538 +0000 UTC m=+2107.571898482" watchObservedRunningTime="2026-11-02 17:57:56.734690386 +0000 UTC m=+2107.575296319"
Nov 02 17:57:57.554697 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.553322    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:57:57.624698 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.624661    4476 generic.go:356] "Generic (PLEG): container finished" podID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerID="c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6" exitCode=0
Nov 02 17:57:57.625074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.624709    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"
Nov 02 17:57:57.625074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.624750    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" event={"ID":"15486176-45b2-4801-97c3-b1e54aa96f5e","Type":"ContainerDied","Data":"c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6"}
Nov 02 17:57:57.625074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.624805    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh" event={"ID":"15486176-45b2-4801-97c3-b1e54aa96f5e","Type":"ContainerDied","Data":"12bc7228fad19aa159c770aa5fe65c81610b83eb90111f091eb802997d32b690"}
Nov 02 17:57:57.625074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.624830    4476 scope.go:122] "RemoveContainer" containerID="c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6"
Nov 02 17:57:57.634989 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.634972    4476 scope.go:122] "RemoveContainer" containerID="c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6"
Nov 02 17:57:57.635250 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:57.635232    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6\": container with ID starting with c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6 not found: ID does not exist" containerID="c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6"
Nov 02 17:57:57.635330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.635258    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6"} err="failed to get container status \"c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6\": rpc error: code = NotFound desc = could not find container \"c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6\": container with ID starting with c2af68f6ace988bdec7a0651ea5af63e6dee4bdd48d633b10c50f01659e159b6 not found: ID does not exist"
Nov 02 17:57:57.733943 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.733906    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734113 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734010    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734113 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734062    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734151    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734201    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734276    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734279    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:57.734359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734319    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734351    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734395    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data\") pod \"15486176-45b2-4801-97c3-b1e54aa96f5e\" (UID: \"15486176-45b2-4801-97c3-b1e54aa96f5e\") "
Nov 02 17:57:57.734570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734397    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:57.734570 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734507    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:57.734769 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.734693    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:57.735089 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.735061    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.735359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.735095    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.735359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.735111    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15486176-45b2-4801-97c3-b1e54aa96f5e-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.735359 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.735124    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.735662 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.735620    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:57.737095 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.737068    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "kube-api-access-fl4w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:57.737304 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.737276    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:57.737385 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.737343    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:57.737439 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.737401    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy" pod "15486176-45b2-4801-97c3-b1e54aa96f5e" (UID: "15486176-45b2-4801-97c3-b1e54aa96f5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:57.836369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.836284    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.836369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.836323    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4w8\" (UniqueName: \"kubernetes.io/projected/15486176-45b2-4801-97c3-b1e54aa96f5e-kube-api-access-fl4w8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.836369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.836335    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.836369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.836345    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.836369 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.836354    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15486176-45b2-4801-97c3-b1e54aa96f5e-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:57.994841 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.994805    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"]
Nov 02 17:57:57.995055 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:57.995029    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" podUID="9e02d380-db11-467f-8d91-f28677dcf9aa" containerName="maas-api" containerID="cri-o://a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21" gracePeriod=30
Nov 02 17:57:58.047456 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.047422    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"]
Nov 02 17:57:58.053248 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.053217    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-8xvlh"]
Nov 02 17:57:58.140345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.139206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:58.287536 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.287505    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:57:58.343141 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.343113    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r\" (UniqueName: \"kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r\") pod \"9e02d380-db11-467f-8d91-f28677dcf9aa\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") "
Nov 02 17:57:58.343330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.343163    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\") pod \"9e02d380-db11-467f-8d91-f28677dcf9aa\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") "
Nov 02 17:57:58.343330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.343208    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs\") pod \"9e02d380-db11-467f-8d91-f28677dcf9aa\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") "
Nov 02 17:57:58.343330 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.343301    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca\") pod \"9e02d380-db11-467f-8d91-f28677dcf9aa\" (UID: \"9e02d380-db11-467f-8d91-f28677dcf9aa\") "
Nov 02 17:57:58.344053 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.344022    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca" pod "9e02d380-db11-467f-8d91-f28677dcf9aa" (UID: "9e02d380-db11-467f-8d91-f28677dcf9aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:58.345897 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.345868    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs" pod "9e02d380-db11-467f-8d91-f28677dcf9aa" (UID: "9e02d380-db11-467f-8d91-f28677dcf9aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:57:58.346581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.346438    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls" pod "9e02d380-db11-467f-8d91-f28677dcf9aa" (UID: "9e02d380-db11-467f-8d91-f28677dcf9aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:57:58.346581 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.346546    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r" pod "9e02d380-db11-467f-8d91-f28677dcf9aa" (UID: "9e02d380-db11-467f-8d91-f28677dcf9aa"). InnerVolumeSpecName "kube-api-access-skb9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:58.448650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.445350    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e02d380-db11-467f-8d91-f28677dcf9aa-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:58.448650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.445387    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skb9r\" (UniqueName: \"kubernetes.io/projected/9e02d380-db11-467f-8d91-f28677dcf9aa-kube-api-access-skb9r\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:58.448650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.445405    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:58.448650 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.445421    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e02d380-db11-467f-8d91-f28677dcf9aa-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:57:58.637526 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.637481    4476 generic.go:356] "Generic (PLEG): container finished" podID="9e02d380-db11-467f-8d91-f28677dcf9aa" containerID="a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21" exitCode=0
Nov 02 17:57:58.638020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.637662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" event={"ID":"9e02d380-db11-467f-8d91-f28677dcf9aa","Type":"ContainerDied","Data":"a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21"}
Nov 02 17:57:58.638020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.637743    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g" event={"ID":"9e02d380-db11-467f-8d91-f28677dcf9aa","Type":"ContainerDied","Data":"3586d1013b8e035f395804a1670e407eba0d83732440af2e642a5037d9246e4d"}
Nov 02 17:57:58.638020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.637783    4476 scope.go:122] "RemoveContainer" containerID="a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21"
Nov 02 17:57:58.638020 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.637978    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"
Nov 02 17:57:58.650819 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.650795    4476 scope.go:122] "RemoveContainer" containerID="a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21"
Nov 02 17:57:58.651143 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:58.651123    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21\": container with ID starting with a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21 not found: ID does not exist" containerID="a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21"
Nov 02 17:57:58.651204 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.651161    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21"} err="failed to get container status \"a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21\": rpc error: code = NotFound desc = could not find container \"a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21\": container with ID starting with a43605953d2cdc76c39f8deeacdc4b5e94ed8a72fb1fa2e134fe6d2ed0778d21 not found: ID does not exist"
Nov 02 17:57:58.802601 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.802507    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"]
Nov 02 17:57:58.805760 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:58.805739    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5bf666-6c48c9684c-sfd8g"]
Nov 02 17:57:59.096753 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:57:59.096586    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:57:59.370478 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.370446    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"]
Nov 02 17:57:59.370705 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.370675    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" containerID="cri-o://2a234e811549dc5f1d2e19a2765d569921877e7d834f4b8086981ec63ca6f8d8" gracePeriod=30
Nov 02 17:57:59.389430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.388372    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" path="/var/lib/kubelet/pods/15486176-45b2-4801-97c3-b1e54aa96f5e/volumes"
Nov 02 17:57:59.389430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.388871    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e02d380-db11-467f-8d91-f28677dcf9aa" path="/var/lib/kubelet/pods/9e02d380-db11-467f-8d91-f28677dcf9aa/volumes"
Nov 02 17:57:59.399903 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.399872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:59.435883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.435853    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"]
Nov 02 17:57:59.436066 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.436047    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" containerID="cri-o://624ec44376df0b442875e9c80a22af60377dada97888bd4e8ad6555a1ef859f6" gracePeriod=30
Nov 02 17:57:59.482907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.482872    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"]
Nov 02 17:57:59.483145 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.483117    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy" containerID="cri-o://944a0e67ecbc5cf0ec80db75da41671ee03f462b818f0502ad5603170d299d13" gracePeriod=30
Nov 02 17:57:59.966619 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:57:59.966586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:00.158117 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:00.158081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:00.865411 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:00.865319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:01.056299 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:01.056262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:01.292154 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:01.292051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:01.803614 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:01.803586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:01.996021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:01.995991    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:02.259309 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:02.259276    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:58:02.259807 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:02.259337    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:58:02.279447 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:02.279419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:02.596346 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:02.596256    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:02.630806 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:02.630776    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 17:58:03.111174 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:03.111143    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:03.944514 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:03.944479    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:03.979785 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:03.979750    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-k4glj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:58:03.979970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:03.979828    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:58:04.139156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.139120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:04.488820 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.488779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:04.712710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.712675    4476 generic.go:356] "Generic (PLEG): container finished" podID="96738341-dadc-4422-8c87-4e0a5c634789" containerID="944a0e67ecbc5cf0ec80db75da41671ee03f462b818f0502ad5603170d299d13" exitCode=0
Nov 02 17:58:04.712874 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.712749    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" event={"ID":"96738341-dadc-4422-8c87-4e0a5c634789","Type":"ContainerDied","Data":"944a0e67ecbc5cf0ec80db75da41671ee03f462b818f0502ad5603170d299d13"}
Nov 02 17:58:04.712874 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.712820    4476 scope.go:122] "RemoveContainer" containerID="bb0721961fc214e4255f946a100603ba98731c86073dc1be72269f296159a7e9"
Nov 02 17:58:04.715008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.714988    4476 generic.go:356] "Generic (PLEG): container finished" podID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerID="624ec44376df0b442875e9c80a22af60377dada97888bd4e8ad6555a1ef859f6" exitCode=0
Nov 02 17:58:04.715168 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.715127    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" event={"ID":"50570585-6b43-4ac9-80a7-df1e86aa8307","Type":"ContainerDied","Data":"624ec44376df0b442875e9c80a22af60377dada97888bd4e8ad6555a1ef859f6"}
Nov 02 17:58:04.717061 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.717041    4476 generic.go:356] "Generic (PLEG): container finished" podID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerID="2a234e811549dc5f1d2e19a2765d569921877e7d834f4b8086981ec63ca6f8d8" exitCode=0
Nov 02 17:58:04.717186 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.717115    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" event={"ID":"b189342d-b7e0-4f17-bbbc-f5202be41650","Type":"ContainerDied","Data":"2a234e811549dc5f1d2e19a2765d569921877e7d834f4b8086981ec63ca6f8d8"}
Nov 02 17:58:04.757321 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.757292    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:58:04.784525 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.784500    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:58:04.820990 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.820962    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:58:04.838592 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838567    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.838748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838618    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.838748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838672    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.838748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838700    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.838748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838728    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.838975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838755    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.838975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838847    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.838975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838857    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.838975 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838947    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.839180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838989    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.839180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.838992    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.839180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839055    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.839180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839087    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.839180 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839136    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.839436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839200    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.839436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839231    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.839436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839263    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.839436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839294    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.839436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839350    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.839436 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839379    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839440    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839475    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839481    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839547    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839597    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839651    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs\") pod \"96738341-dadc-4422-8c87-4e0a5c634789\" (UID: \"96738341-dadc-4422-8c87-4e0a5c634789\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839692    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839721    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.839948 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.839781    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket\") pod \"50570585-6b43-4ac9-80a7-df1e86aa8307\" (UID: \"50570585-6b43-4ac9-80a7-df1e86aa8307\") "
Nov 02 17:58:04.840430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.840308    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.840430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.840326    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.840430 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.840341    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.840582 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.840541    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.841620 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.841592    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:04.841840 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.841822    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.841936 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.841819    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.842719 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.842123    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:04.842719 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.842239    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.842719 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.842501    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.842719 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.842566    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.843018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.842752    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.843018 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.842824    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.843148 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843024    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.843148 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843046    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:04.843261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843150    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:04.843261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843160    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "kube-api-access-6qbhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.843261 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843256    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.843515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843466    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.843515 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843501    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket" pod "96738341-dadc-4422-8c87-4e0a5c634789" (UID: "96738341-dadc-4422-8c87-4e0a5c634789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.844092 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843526    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "kube-api-access-hpkts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.844092 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843545    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:04.844092 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.843962    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.844335 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.844311    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "kube-api-access-z2znr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:04.845190 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.845170    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.845780 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.845764    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy" pod "50570585-6b43-4ac9-80a7-df1e86aa8307" (UID: "50570585-6b43-4ac9-80a7-df1e86aa8307"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.941654    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo\") pod \"b189342d-b7e0-4f17-bbbc-f5202be41650\" (UID: \"b189342d-b7e0-4f17-bbbc-f5202be41650\") "
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942524    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942551    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942568    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2znr\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-kube-api-access-z2znr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942583    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942596    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942609    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96738341-dadc-4422-8c87-4e0a5c634789-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942623    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942655    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50570585-6b43-4ac9-80a7-df1e86aa8307-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942667    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942681    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942695    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b189342d-b7e0-4f17-bbbc-f5202be41650-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942709    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96738341-dadc-4422-8c87-4e0a5c634789-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942723    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942736    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942749    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942763    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96738341-dadc-4422-8c87-4e0a5c634789-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942776    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942791    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpkts\" (UniqueName: \"kubernetes.io/projected/b189342d-b7e0-4f17-bbbc-f5202be41650-kube-api-access-hpkts\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942805    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942821    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50570585-6b43-4ac9-80a7-df1e86aa8307-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942835    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qbhn\" (UniqueName: \"kubernetes.io/projected/96738341-dadc-4422-8c87-4e0a5c634789-kube-api-access-6qbhn\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942849    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50570585-6b43-4ac9-80a7-df1e86aa8307-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.942940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.942864    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:04.944983 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:04.944945    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo" pod "b189342d-b7e0-4f17-bbbc-f5202be41650" (UID: "b189342d-b7e0-4f17-bbbc-f5202be41650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:05.044424 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.044390    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b189342d-b7e0-4f17-bbbc-f5202be41650-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:05.293835 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.293746    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:05.725047 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.725014    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"
Nov 02 17:58:05.725272 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.725055    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx" event={"ID":"b189342d-b7e0-4f17-bbbc-f5202be41650","Type":"ContainerDied","Data":"819a33964c661854941aab07b8f4e0a6a1d92d090eeeb106367ed8d56485084a"}
Nov 02 17:58:05.725272 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.725124    4476 scope.go:122] "RemoveContainer" containerID="2a234e811549dc5f1d2e19a2765d569921877e7d834f4b8086981ec63ca6f8d8"
Nov 02 17:58:05.727172 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.727139    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"
Nov 02 17:58:05.727301 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.727187    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r" event={"ID":"96738341-dadc-4422-8c87-4e0a5c634789","Type":"ContainerDied","Data":"f5182f2d30526fe971a90462cbd630e1469b9bca4728a0c4fce6a2d74f1385bc"}
Nov 02 17:58:05.728996 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.728975    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj" event={"ID":"50570585-6b43-4ac9-80a7-df1e86aa8307","Type":"ContainerDied","Data":"c18ddbf24bf8790149dde970d8d2b1c817a44e12afc1bc9da21950221020af98"}
Nov 02 17:58:05.729119 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.729009    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"
Nov 02 17:58:05.736827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.736810    4476 scope.go:122] "RemoveContainer" containerID="944a0e67ecbc5cf0ec80db75da41671ee03f462b818f0502ad5603170d299d13"
Nov 02 17:58:05.746748 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.746724    4476 scope.go:122] "RemoveContainer" containerID="624ec44376df0b442875e9c80a22af60377dada97888bd4e8ad6555a1ef859f6"
Nov 02 17:58:05.890855 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.890821    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"]
Nov 02 17:58:05.940775 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:05.940743    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffxmx"]
Nov 02 17:58:06.065473 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:06.065399    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"]
Nov 02 17:58:06.074832 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:06.074808    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8jc9r"]
Nov 02 17:58:06.185422 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:06.185391    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"]
Nov 02 17:58:06.189150 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:06.189124    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-k4glj"]
Nov 02 17:58:06.613189 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:06.613149    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:07.383232 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:07.383191    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" path="/var/lib/kubelet/pods/50570585-6b43-4ac9-80a7-df1e86aa8307/volumes"
Nov 02 17:58:07.383843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:07.383823    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96738341-dadc-4422-8c87-4e0a5c634789" path="/var/lib/kubelet/pods/96738341-dadc-4422-8c87-4e0a5c634789/volumes"
Nov 02 17:58:07.384309 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:07.384289    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" path="/var/lib/kubelet/pods/b189342d-b7e0-4f17-bbbc-f5202be41650/volumes"
Nov 02 17:58:07.386812 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:07.386791    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:08.512167 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:08.512113    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:09.033883 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.033841    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"]
Nov 02 17:58:09.034402 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034386    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9e02d380-db11-467f-8d91-f28677dcf9aa" containerName="maas-api"
Nov 02 17:58:09.034453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034405    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e02d380-db11-467f-8d91-f28677dcf9aa" containerName="maas-api"
Nov 02 17:58:09.034453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034418    4476 cpu_manager.go:414] "containerMap: removing container" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy"
Nov 02 17:58:09.034453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034424    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy"
Nov 02 17:58:09.034453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034437    4476 cpu_manager.go:414] "containerMap: removing container" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy"
Nov 02 17:58:09.034453 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034443    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034466    4476 cpu_manager.go:414] "containerMap: removing container" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034473    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034487    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034493    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034501    4476 cpu_manager.go:414] "containerMap: removing container" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034506    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034603    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e02d380-db11-467f-8d91-f28677dcf9aa" containerName="maas-api"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034612    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034619    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="96738341-dadc-4422-8c87-4e0a5c634789" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034645    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="50570585-6b43-4ac9-80a7-df1e86aa8307" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034653    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b189342d-b7e0-4f17-bbbc-f5202be41650" containerName="istio-proxy"
Nov 02 17:58:09.034654 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.034660    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="15486176-45b2-4801-97c3-b1e54aa96f5e" containerName="istio-proxy"
Nov 02 17:58:09.040228 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.040197    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.090364 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.090329    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.090742 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.090686    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.090929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.090912    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256vh\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.091076 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.091051    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.091156 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.091122    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.091217 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.091150    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.091271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.091238    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.091351 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.091334    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.091408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.091362    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.095977 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.095947    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f2289f-openshift-default-dockercfg-n55s9\""
Nov 02 17:58:09.146934 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.146899    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"]
Nov 02 17:58:09.179106 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.179074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:09.199212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.198952    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199065    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199090    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199137    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199212 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199182    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199213    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256vh\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199276    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199339    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.199737 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199365    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.200008 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.199867    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.200205 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.200086    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.200602 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.200577    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.201220 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.201198    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.201418 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.201400    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.205787 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.205758    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.206931 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.206910    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.265399 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.265361    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.265553 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.265506    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-256vh\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh\") pod \"e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") " pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.376676 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.376639    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:09.436710 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.436674    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:09.830268 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.830240    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"]
Nov 02 17:58:09.831447 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:58:09.831420    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7247c4_37c2_4e47_bb04_fa7d7d96b9eb.slice/crio-8b49fcc46583d6fbc2abfaf3e5056dca0ba248c00516052d2abb6912cbb39d71 WatchSource:0}: Error finding container 8b49fcc46583d6fbc2abfaf3e5056dca0ba248c00516052d2abb6912cbb39d71: Status 404 returned error can't find the container with id 8b49fcc46583d6fbc2abfaf3e5056dca0ba248c00516052d2abb6912cbb39d71
Nov 02 17:58:09.834112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.834083    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:58:09.834182 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.834168    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:58:09.834220 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:09.834201    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 17:58:10.751414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:10.751379    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"]
Nov 02 17:58:10.771293 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:10.771255    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" event={"ID":"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb","Type":"ContainerStarted","Data":"5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"}
Nov 02 17:58:10.771493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:10.771314    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" event={"ID":"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb","Type":"ContainerStarted","Data":"8b49fcc46583d6fbc2abfaf3e5056dca0ba248c00516052d2abb6912cbb39d71"}
Nov 02 17:58:10.858940 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:10.858873    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"]
Nov 02 17:58:11.107450 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.107338    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podStartSLOduration=3.107324987 podStartE2EDuration="3.107324987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:11.107016234 +0000 UTC m=+2121.947622169" watchObservedRunningTime="2026-11-02 17:58:11.107324987 +0000 UTC m=+2121.947930921"
Nov 02 17:58:11.169408 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.169186    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:11.369506 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.369468    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:11.385574 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.385548    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:11.385715 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.385650    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:11.597251 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.597222    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:11.778947 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.778872    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:11.779952 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:11.779932    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 17:58:12.104793 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:12.104708    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:12.691960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:12.691913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:14.071226 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:14.071187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:14.482761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:14.482729    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:15.752021 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:15.751982    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd" podUID="a0dec886-06b7-4f44-b095-ab8018293f93" containerName="payload-processing" containerID="cri-o://6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412" gracePeriod=25
Nov 02 17:58:15.859791 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:15.859754    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t" podUID="d826bdb0-881e-4479-9cb6-361a583919bf" containerName="payload-pre-processing" containerID="cri-o://faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa" gracePeriod=25
Nov 02 17:58:16.022886 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.022570    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:58:16.076124 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.076084    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm\" (UniqueName: \"kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm\") pod \"a0dec886-06b7-4f44-b095-ab8018293f93\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") "
Nov 02 17:58:16.076295 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.076262    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume\") pod \"a0dec886-06b7-4f44-b095-ab8018293f93\" (UID: \"a0dec886-06b7-4f44-b095-ab8018293f93\") "
Nov 02 17:58:16.076909 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.076876    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume" pod "a0dec886-06b7-4f44-b095-ab8018293f93" (UID: "a0dec886-06b7-4f44-b095-ab8018293f93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:16.078858 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.078825    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm" pod "a0dec886-06b7-4f44-b095-ab8018293f93" (UID: "a0dec886-06b7-4f44-b095-ab8018293f93"). InnerVolumeSpecName "kube-api-access-lwwwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:16.118142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.118115    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:58:16.177518 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.177471    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume\") pod \"d826bdb0-881e-4479-9cb6-361a583919bf\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") "
Nov 02 17:58:16.177729 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.177575    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8\" (UniqueName: \"kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8\") pod \"d826bdb0-881e-4479-9cb6-361a583919bf\" (UID: \"d826bdb0-881e-4479-9cb6-361a583919bf\") "
Nov 02 17:58:16.177924 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.177891    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume" pod "d826bdb0-881e-4479-9cb6-361a583919bf" (UID: "d826bdb0-881e-4479-9cb6-361a583919bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:16.178140 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.178118    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dec886-06b7-4f44-b095-ab8018293f93-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:16.178215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.178145    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d826bdb0-881e-4479-9cb6-361a583919bf-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:16.178215 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.178158    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwwm\" (UniqueName: \"kubernetes.io/projected/a0dec886-06b7-4f44-b095-ab8018293f93-kube-api-access-lwwwm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:16.179992 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.179962    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8" pod "d826bdb0-881e-4479-9cb6-361a583919bf" (UID: "d826bdb0-881e-4479-9cb6-361a583919bf"). InnerVolumeSpecName "kube-api-access-dx4z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:16.278915 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.278832    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4z8\" (UniqueName: \"kubernetes.io/projected/d826bdb0-881e-4479-9cb6-361a583919bf-kube-api-access-dx4z8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 17:58:16.815393 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.815351    4476 generic.go:356] "Generic (PLEG): container finished" podID="a0dec886-06b7-4f44-b095-ab8018293f93" containerID="6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412" exitCode=0
Nov 02 17:58:16.815834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.815425    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"
Nov 02 17:58:16.815834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.815438    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd" event={"ID":"a0dec886-06b7-4f44-b095-ab8018293f93","Type":"ContainerDied","Data":"6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412"}
Nov 02 17:58:16.815834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.815490    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd" event={"ID":"a0dec886-06b7-4f44-b095-ab8018293f93","Type":"ContainerDied","Data":"c519d536a12b3f0e5f569e8cac05f7d5f820d519c264bae359768d26442d3f94"}
Nov 02 17:58:16.815834 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.815514    4476 scope.go:122] "RemoveContainer" containerID="6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412"
Nov 02 17:58:16.817307 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.817282    4476 generic.go:356] "Generic (PLEG): container finished" podID="d826bdb0-881e-4479-9cb6-361a583919bf" containerID="faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa" exitCode=0
Nov 02 17:58:16.817401 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.817379    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"
Nov 02 17:58:16.817401 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.817389    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t" event={"ID":"d826bdb0-881e-4479-9cb6-361a583919bf","Type":"ContainerDied","Data":"faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa"}
Nov 02 17:58:16.817489 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.817434    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t" event={"ID":"d826bdb0-881e-4479-9cb6-361a583919bf","Type":"ContainerDied","Data":"ab536a368bfae721e9140308051a3fb7402b61c29f45eb13a4c1bb6489c55984"}
Nov 02 17:58:16.830485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.826890    4476 scope.go:122] "RemoveContainer" containerID="6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412"
Nov 02 17:58:16.830485 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:58:16.827273    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412\": container with ID starting with 6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412 not found: ID does not exist" containerID="6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412"
Nov 02 17:58:16.830485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.827303    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412"} err="failed to get container status \"6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412\": rpc error: code = NotFound desc = could not find container \"6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412\": container with ID starting with 6fefeda596f67d45c602666d151bc8d70421a6dccdd7af57c6d74c625863f412 not found: ID does not exist"
Nov 02 17:58:16.830485 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.827325    4476 scope.go:122] "RemoveContainer" containerID="faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa"
Nov 02 17:58:16.840059 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.840040    4476 scope.go:122] "RemoveContainer" containerID="faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa"
Nov 02 17:58:16.840384 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:58:16.840363    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa\": container with ID starting with faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa not found: ID does not exist" containerID="faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa"
Nov 02 17:58:16.840432 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.840391    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa"} err="failed to get container status \"faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa\": rpc error: code = NotFound desc = could not find container \"faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa\": container with ID starting with faa6fd2758c09426e0e8e4b6172c81bfb01480297b1bdbe8a06928ef083c69aa not found: ID does not exist"
Nov 02 17:58:16.982474 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:16.982444    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"]
Nov 02 17:58:17.032132 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:17.032046    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1cea7c-56dcf5599f-96z5t"]
Nov 02 17:58:17.136545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:17.136511    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"]
Nov 02 17:58:17.154127 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:17.154099    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cea7c-9fb6d5f89-xcmkd"]
Nov 02 17:58:17.383768 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:17.383739    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dec886-06b7-4f44-b095-ab8018293f93" path="/var/lib/kubelet/pods/a0dec886-06b7-4f44-b095-ab8018293f93/volumes"
Nov 02 17:58:17.384143 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:17.384130    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d826bdb0-881e-4479-9cb6-361a583919bf" path="/var/lib/kubelet/pods/d826bdb0-881e-4479-9cb6-361a583919bf/volumes"
Nov 02 17:58:17.633246 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:17.633217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:18.933486 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:18.933451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:19.459900 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:19.459863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:19.701471 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:19.701402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:22.011258 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:22.011217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:23.773528 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:23.773478    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:27.115374 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:27.115342    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:27.319311 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:27.319259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:28.835586 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:28.835551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:29.060440 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:58:29.060394    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:58:29.072242 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:29.072206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:29.358604 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:29.358567    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:30.930403 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:30.930371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:31.586535 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:31.586491    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:31.795222 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:31.795187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:35.805440 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:35.805405    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:36.023111 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:36.023080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:36.244932 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:36.244898    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:39.283353 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:39.283320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:39.489356 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:39.489327    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:41.269905 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:41.269864    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:46.384402 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:46.384365    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:46.655444 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:46.655361    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:47.801935 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:47.801897    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:48.407120 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:48.407091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:48.686896 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:48.686811    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:51.569370 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:51.569331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:52.010735 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:52.010701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:52.276793 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:52.276705    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:52.490326 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:52.490295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:55.341384 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.341348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:55.535609 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.535579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:55.888583 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.888550    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"]
Nov 02 17:58:55.889368 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.889346    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d826bdb0-881e-4479-9cb6-361a583919bf" containerName="payload-pre-processing"
Nov 02 17:58:55.889438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.889371    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d826bdb0-881e-4479-9cb6-361a583919bf" containerName="payload-pre-processing"
Nov 02 17:58:55.889438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.889398    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a0dec886-06b7-4f44-b095-ab8018293f93" containerName="payload-processing"
Nov 02 17:58:55.889438 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.889405    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dec886-06b7-4f44-b095-ab8018293f93" containerName="payload-processing"
Nov 02 17:58:55.889545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.889534    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d826bdb0-881e-4479-9cb6-361a583919bf" containerName="payload-pre-processing"
Nov 02 17:58:55.889579 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.889549    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dec886-06b7-4f44-b095-ab8018293f93" containerName="payload-processing"
Nov 02 17:58:55.895112 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.895079    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:55.960147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.960114    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:55.960147 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.960154    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfh7z\" (UniqueName: \"kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z\") pod \"payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:55.996827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.996783    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f42381-dockercfg-cfprf\""
Nov 02 17:58:55.996827 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:55.996799    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f42381\""
Nov 02 17:58:56.000183 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.000161    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"]
Nov 02 17:58:56.061843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.061808    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:56.061843 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.061850    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfh7z\" (UniqueName: \"kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z\") pod \"payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:56.062502 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.062483    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:56.118943 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.118908    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfh7z\" (UniqueName: \"kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z\") pod \"payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:56.209873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.209774    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:56.615406 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:56.615382    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"]
Nov 02 17:58:56.617595 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:58:56.617568    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2905fa28_45e0_4145_9d16_d008e84ce6da.slice/crio-893d582fbe0ad75cd3e8175153a5a3e4db6139084a0ae702ac7adcc55bc10b8b WatchSource:0}: Error finding container 893d582fbe0ad75cd3e8175153a5a3e4db6139084a0ae702ac7adcc55bc10b8b: Status 404 returned error can't find the container with id 893d582fbe0ad75cd3e8175153a5a3e4db6139084a0ae702ac7adcc55bc10b8b
Nov 02 17:58:57.097960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:57.097862    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s" event={"ID":"2905fa28-45e0-4145-9d16-d008e84ce6da","Type":"ContainerStarted","Data":"b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3"}
Nov 02 17:58:57.097960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:57.097921    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s" event={"ID":"2905fa28-45e0-4145-9d16-d008e84ce6da","Type":"ContainerStarted","Data":"893d582fbe0ad75cd3e8175153a5a3e4db6139084a0ae702ac7adcc55bc10b8b"}
Nov 02 17:58:57.097960 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:57.097941    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:58:57.111974 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:57.111925    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:57.215271 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:58:57.215221    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s" podStartSLOduration=2.21520293 podStartE2EDuration="2.21520293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:57.212174478 +0000 UTC m=+2168.052780423" watchObservedRunningTime="2026-11-02 17:58:57.21520293 +0000 UTC m=+2168.055808865"
Nov 02 17:58:59.060588 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:58:59.060550    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:59:00.500414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:00.500373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:01.605873 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:01.605837    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:01.820559 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:01.820529    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:02.364160 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:02.364119    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:04.506699 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.506667    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"]
Nov 02 17:59:04.510568 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.510549    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.613998 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.613967    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"]
Nov 02 17:59:04.616209 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.616189    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ccf013\"/\"openshift-service-ca.crt\""
Nov 02 17:59:04.616281 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.616236    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ccf013\"/\"kube-root-ca.crt\""
Nov 02 17:59:04.616281 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.616249    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ccf013\"/\"ipp-route-ccf013-kserve-self-signed-certs\""
Nov 02 17:59:04.616362 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.616282    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ccf013\"/\"default-dockercfg-tbw8c\""
Nov 02 17:59:04.659681 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.659598    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.659907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.659732    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.659907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.659786    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.659907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.659879    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.660090 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.659942    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ph2\" (UniqueName: \"kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.660090 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.660020    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.669329 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.669100    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:04.760761 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.760666    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ph2\" (UniqueName: \"kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.760929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.760776    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.760929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.760837    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.760929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.760873    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.760929 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.760907    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.761142 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.760963    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.761275 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.761249    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.761345 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.761332    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.761557 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.761535    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.763884 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.763862    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.766167 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.766143    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.817918 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.817885    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ph2\" (UniqueName: \"kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2\") pod \"ipp-route-ccf013-kserve-5974b5b445-j45wh\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") " pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:04.823561 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:04.823540    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:05.221875 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:05.221849    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"]
Nov 02 17:59:05.223293 ip-10-0-136-192 kubenswrapper[4476]: W1102 17:59:05.223263    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b99fb76_c5c0_4fba_a62b_ae1d7e4104da.slice/crio-04c0f24ebae8c8bc0d5d2a22ff954e3863dc971708b6931ddd960908b3898b65 WatchSource:0}: Error finding container 04c0f24ebae8c8bc0d5d2a22ff954e3863dc971708b6931ddd960908b3898b65: Status 404 returned error can't find the container with id 04c0f24ebae8c8bc0d5d2a22ff954e3863dc971708b6931ddd960908b3898b65
Nov 02 17:59:06.162067 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:06.162021    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh" event={"ID":"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da","Type":"ContainerStarted","Data":"d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5"}
Nov 02 17:59:06.162475 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:06.162078    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh" event={"ID":"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da","Type":"ContainerStarted","Data":"04c0f24ebae8c8bc0d5d2a22ff954e3863dc971708b6931ddd960908b3898b65"}
Nov 02 17:59:06.283203 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:06.283153    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh" podStartSLOduration=2.283137655 podStartE2EDuration="2.283137655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:06.279083028 +0000 UTC m=+2177.119688964" watchObservedRunningTime="2026-11-02 17:59:06.283137655 +0000 UTC m=+2177.123743567"
Nov 02 17:59:06.824388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:06.824354    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:06.871259 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:06.871222    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:08.107155 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:08.107125    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 17:59:08.806210 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:08.806176    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:09.375414 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:09.375369    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:09.572033 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:09.571998    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:11.194354 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:11.194321    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:11.667406 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:11.667357    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:12.636447 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:12.636415    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:13.254970 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:13.254917    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:13.771901 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:13.771861    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:14.049911 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:14.049818    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:14.252792 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:14.252752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:14.458740 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:14.458703    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:14.824290 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:14.824202    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:14.837249 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:14.837223    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:15.244035 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:15.244007    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 17:59:17.323328 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:17.323291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:17.538749 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:17.538714    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:17.818221 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:17.818188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:19.482163 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:19.482079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:22.648545 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:22.648506    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:22.840907 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:22.840872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:24.821670 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:24.821602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:29.062958 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:59:29.062913    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 17:59:30.301286 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:30.301092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:36.619487 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:36.619455    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:36.803427 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:36.803389    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:41.577493 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:41.577458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:46.458074 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:46.457998    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:51.263388 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:51.263348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:52.349367 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:52.349329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:56.060655 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:56.059132    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:56.283398 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:56.283370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:56.582508 ip-10-0-136-192 kubenswrapper[4476]: I1102 17:59:56.582481    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:59.073901 ip-10-0-136-192 kubenswrapper[4476]: E1102 17:59:59.073827    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:00:00.169037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.169000    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"]
Nov 02 18:00:00.173258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.173233    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.187878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.187845    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"]
Nov 02 18:00:00.191905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.191880    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"]
Nov 02 18:00:00.192059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.192022    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.196557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.196533    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.218060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.218028    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"]
Nov 02 18:00:00.221797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.221777    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.229855 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.229828    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"]
Nov 02 18:00:00.240713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.231973    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:00:00.240713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.235274    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.277368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.277331    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"]
Nov 02 18:00:00.281012 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.280980    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"]
Nov 02 18:00:00.316466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.316427    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"]
Nov 02 18:00:00.329150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.329100    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"]
Nov 02 18:00:00.331359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331299    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqckf\" (UniqueName: \"kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.331563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331367    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.331563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331402    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv9lb\" (UniqueName: \"kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.331563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331464    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr28r\" (UniqueName: \"kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r\") pod \"maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.331563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331508    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.331563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331555    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.332035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331594    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.332035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331669    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.332035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331691    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvmp\" (UniqueName: \"kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp\") pod \"maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.332035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.331716    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.392325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.392289    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"]
Nov 02 18:00:00.434397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434298    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.434397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434366    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.434663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434431    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.434663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434459    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvmp\" (UniqueName: \"kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp\") pod \"maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.434663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434485    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.434663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434578    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqckf\" (UniqueName: \"kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.434892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434668    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.434892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434700    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv9lb\" (UniqueName: \"kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.434892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434772    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr28r\" (UniqueName: \"kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r\") pod \"maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.434892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.434823    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.435116 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.435092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.435293 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.435267    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.435527 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.435506    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.436131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.436049    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.436238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.436134    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.492907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.492823    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv9lb\" (UniqueName: \"kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.493180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.493153    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqckf\" (UniqueName: \"kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf\") pod \"maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.493428 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.493402    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr28r\" (UniqueName: \"kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r\") pod \"maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.493556 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.493537    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvmp\" (UniqueName: \"kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp\") pod \"maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.493707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.493692    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.508937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.508867    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:00.516862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.516832    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:00.550875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.550832    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:00.569009 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.568685    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:00.791980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.791930    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:00.912413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.912383    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"]
Nov 02 18:00:00.915470 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:00.915400    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd086c8bf_5d20_48d4_9eec_5fcad95c40a8.slice/crio-e37d8467ebe2c85ed47c91b172732bbd1dbaf2ec62df9a98b57b469eaa0ec6a4 WatchSource:0}: Error finding container e37d8467ebe2c85ed47c91b172732bbd1dbaf2ec62df9a98b57b469eaa0ec6a4: Status 404 returned error can't find the container with id e37d8467ebe2c85ed47c91b172732bbd1dbaf2ec62df9a98b57b469eaa0ec6a4
Nov 02 18:00:00.977590 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:00.977514    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"]
Nov 02 18:00:00.986799 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:00.986761    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d26c76a_495a_4fb7_9caa_5e05c8d3f912.slice/crio-3c81d8d63137af27d940e3e58f1ade5513c53ed856c624e3d831a400cbc00eb9 WatchSource:0}: Error finding container 3c81d8d63137af27d940e3e58f1ade5513c53ed856c624e3d831a400cbc00eb9: Status 404 returned error can't find the container with id 3c81d8d63137af27d940e3e58f1ade5513c53ed856c624e3d831a400cbc00eb9
Nov 02 18:00:01.020730 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.020696    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"]
Nov 02 18:00:01.024113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.024087    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"]
Nov 02 18:00:01.025972 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:01.025946    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb519242a_cca6_4bc3_afb8_fc718466eca9.slice/crio-c708f8e19476d9cef91c232d8132498ad61857ac37e64a1a746129bad7c45b9a WatchSource:0}: Error finding container c708f8e19476d9cef91c232d8132498ad61857ac37e64a1a746129bad7c45b9a: Status 404 returned error can't find the container with id c708f8e19476d9cef91c232d8132498ad61857ac37e64a1a746129bad7c45b9a
Nov 02 18:00:01.026236 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:01.026219    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a9de3c_e58f_45f5_bca2_3db421ec9a80.slice/crio-e24476136b070c15a9b116db63e5ed4036e9eb48cdd1bd6be8ac2b25cc3f9ed1 WatchSource:0}: Error finding container e24476136b070c15a9b116db63e5ed4036e9eb48cdd1bd6be8ac2b25cc3f9ed1: Status 404 returned error can't find the container with id e24476136b070c15a9b116db63e5ed4036e9eb48cdd1bd6be8ac2b25cc3f9ed1
Nov 02 18:00:01.179858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.179832    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"]
Nov 02 18:00:01.182599 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:01.182565    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7d6fa8_3f58_4594_98f4_59eff2764480.slice/crio-b71da98555a85a9f9e88cabdf309902273fa8e6274dd8bad228a5313d2b13eb2 WatchSource:0}: Error finding container b71da98555a85a9f9e88cabdf309902273fa8e6274dd8bad228a5313d2b13eb2: Status 404 returned error can't find the container with id b71da98555a85a9f9e88cabdf309902273fa8e6274dd8bad228a5313d2b13eb2
Nov 02 18:00:01.350078 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.350049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:01.607703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.607665    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerStarted","Data":"2b918a5e6d1d9b81caa56c5fa50cce67024d28008f0866be27c02e0f16368485"}
Nov 02 18:00:01.607900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.607733    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerStarted","Data":"3c81d8d63137af27d940e3e58f1ade5513c53ed856c624e3d831a400cbc00eb9"}
Nov 02 18:00:01.609388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.609357    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerStarted","Data":"e24476136b070c15a9b116db63e5ed4036e9eb48cdd1bd6be8ac2b25cc3f9ed1"}
Nov 02 18:00:01.611201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.611133    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerStarted","Data":"413c9285036bdbb6be4870e445cbf010968b8f08008fbe2b8e386a9e87ed5b39"}
Nov 02 18:00:01.611201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.611195    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerStarted","Data":"e37d8467ebe2c85ed47c91b172732bbd1dbaf2ec62df9a98b57b469eaa0ec6a4"}
Nov 02 18:00:01.613021 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.612991    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerStarted","Data":"e57fefefc653b277410d7703ce0340b44a89107883217df7bfebb383fb790052"}
Nov 02 18:00:01.613130 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.613037    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerStarted","Data":"b71da98555a85a9f9e88cabdf309902273fa8e6274dd8bad228a5313d2b13eb2"}
Nov 02 18:00:01.614383 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.614359    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerStarted","Data":"c708f8e19476d9cef91c232d8132498ad61857ac37e64a1a746129bad7c45b9a"}
Nov 02 18:00:01.726942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.726870    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" podStartSLOduration=1.7268515089999998 podStartE2EDuration="1.726851509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.721206166 +0000 UTC m=+2232.561812099" watchObservedRunningTime="2026-11-02 18:00:01.726851509 +0000 UTC m=+2232.567457446"
Nov 02 18:00:01.836328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.836222    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" podStartSLOduration=1.836206998 podStartE2EDuration="1.836206998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.834665878 +0000 UTC m=+2232.675271802" watchObservedRunningTime="2026-11-02 18:00:01.836206998 +0000 UTC m=+2232.676812932"
Nov 02 18:00:01.953192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:01.953126    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" podStartSLOduration=1.953106024 podStartE2EDuration="1.953106024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.947862595 +0000 UTC m=+2232.788468529" watchObservedRunningTime="2026-11-02 18:00:01.953106024 +0000 UTC m=+2232.793711963"
Nov 02 18:00:02.623577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.623538    4476 generic.go:356] "Generic (PLEG): container finished" podID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerID="5b5a70f89f86faa8a451816f1433e2d345af85f4022cb5643c3345cd37eaceca" exitCode=60
Nov 02 18:00:02.624101 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.623653    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerDied","Data":"5b5a70f89f86faa8a451816f1433e2d345af85f4022cb5643c3345cd37eaceca"}
Nov 02 18:00:02.624101 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.623975    4476 scope.go:122] "RemoveContainer" containerID="5b5a70f89f86faa8a451816f1433e2d345af85f4022cb5643c3345cd37eaceca"
Nov 02 18:00:02.626125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.626021    4476 generic.go:356] "Generic (PLEG): container finished" podID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerID="413c9285036bdbb6be4870e445cbf010968b8f08008fbe2b8e386a9e87ed5b39" exitCode=60
Nov 02 18:00:02.626125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.626055    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerDied","Data":"413c9285036bdbb6be4870e445cbf010968b8f08008fbe2b8e386a9e87ed5b39"}
Nov 02 18:00:02.626356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.626339    4476 scope.go:122] "RemoveContainer" containerID="413c9285036bdbb6be4870e445cbf010968b8f08008fbe2b8e386a9e87ed5b39"
Nov 02 18:00:02.628041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.628018    4476 generic.go:356] "Generic (PLEG): container finished" podID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerID="e57fefefc653b277410d7703ce0340b44a89107883217df7bfebb383fb790052" exitCode=60
Nov 02 18:00:02.628180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.628162    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerDied","Data":"e57fefefc653b277410d7703ce0340b44a89107883217df7bfebb383fb790052"}
Nov 02 18:00:02.628288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.628272    4476 scope.go:122] "RemoveContainer" containerID="e57fefefc653b277410d7703ce0340b44a89107883217df7bfebb383fb790052"
Nov 02 18:00:02.630822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.630800    4476 generic.go:356] "Generic (PLEG): container finished" podID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerID="1004f89c80b41e7125fae3c4b9bcd75fb7c646b71bbc21e2bd61e610db05f6f8" exitCode=60
Nov 02 18:00:02.631069 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.631052    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerDied","Data":"1004f89c80b41e7125fae3c4b9bcd75fb7c646b71bbc21e2bd61e610db05f6f8"}
Nov 02 18:00:02.631184 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.631166    4476 scope.go:122] "RemoveContainer" containerID="1004f89c80b41e7125fae3c4b9bcd75fb7c646b71bbc21e2bd61e610db05f6f8"
Nov 02 18:00:02.632614 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.632591    4476 generic.go:356] "Generic (PLEG): container finished" podID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerID="2b918a5e6d1d9b81caa56c5fa50cce67024d28008f0866be27c02e0f16368485" exitCode=60
Nov 02 18:00:02.632711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.632662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerDied","Data":"2b918a5e6d1d9b81caa56c5fa50cce67024d28008f0866be27c02e0f16368485"}
Nov 02 18:00:02.632981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:02.632964    4476 scope.go:122] "RemoveContainer" containerID="2b918a5e6d1d9b81caa56c5fa50cce67024d28008f0866be27c02e0f16368485"
Nov 02 18:00:03.645005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.644508    4476 generic.go:356] "Generic (PLEG): container finished" podID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerID="5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863" exitCode=0
Nov 02 18:00:03.645005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.644591    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" event={"ID":"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb","Type":"ContainerDied","Data":"5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"}
Nov 02 18:00:03.645415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.645081    4476 scope.go:122] "RemoveContainer" containerID="5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"
Nov 02 18:00:03.647863 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.647830    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:00:03.647988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.647918    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:00:03.647988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.647962    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:00:03.647988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.647974    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerStarted","Data":"0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d"}
Nov 02 18:00:03.651257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.651216    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerStarted","Data":"d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b"}
Nov 02 18:00:03.653695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.653670    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerStarted","Data":"3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa"}
Nov 02 18:00:03.655695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.655671    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerStarted","Data":"9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6"}
Nov 02 18:00:03.657926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.657907    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerStarted","Data":"297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897"}
Nov 02 18:00:03.999243 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:03.999190    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" podStartSLOduration=3.999177292 podStartE2EDuration="3.999177292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.995440618 +0000 UTC m=+2234.836046552" watchObservedRunningTime="2026-11-02 18:00:03.999177292 +0000 UTC m=+2234.839783226"
Nov 02 18:00:04.360899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.360837    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" podStartSLOduration=4.360816821 podStartE2EDuration="4.360816821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.35808336 +0000 UTC m=+2235.198689294" watchObservedRunningTime="2026-11-02 18:00:04.360816821 +0000 UTC m=+2235.201422756"
Nov 02 18:00:04.679449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.679349    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" event={"ID":"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb","Type":"ContainerStarted","Data":"d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79"}
Nov 02 18:00:04.681888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.681858    4476 generic.go:356] "Generic (PLEG): container finished" podID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerID="0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d" exitCode=60
Nov 02 18:00:04.682052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.681888    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerDied","Data":"0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d"}
Nov 02 18:00:04.682052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.681954    4476 scope.go:122] "RemoveContainer" containerID="5b5a70f89f86faa8a451816f1433e2d345af85f4022cb5643c3345cd37eaceca"
Nov 02 18:00:04.682199 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.682185    4476 scope.go:122] "RemoveContainer" containerID="0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d"
Nov 02 18:00:04.682504 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:04.682482    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f_odh-ai-gateway-infra(23a9de3c-e58f-45f5-bca2-3db421ec9a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80"
Nov 02 18:00:04.684279 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.684255    4476 generic.go:356] "Generic (PLEG): container finished" podID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerID="d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b" exitCode=60
Nov 02 18:00:04.684409 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.684333    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerDied","Data":"d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b"}
Nov 02 18:00:04.684609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.684592    4476 scope.go:122] "RemoveContainer" containerID="d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b"
Nov 02 18:00:04.684910 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:04.684885    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z_odh-ai-gateway-infra(d086c8bf-5d20-48d4-9eec-5fcad95c40a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8"
Nov 02 18:00:04.686777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.686751    4476 generic.go:356] "Generic (PLEG): container finished" podID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerID="3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa" exitCode=60
Nov 02 18:00:04.686905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.686829    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerDied","Data":"3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa"}
Nov 02 18:00:04.686992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.686976    4476 scope.go:122] "RemoveContainer" containerID="3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa"
Nov 02 18:00:04.687202 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:04.687175    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx_odh-ai-gateway-infra(bf7d6fa8-3f58-4594-98f4-59eff2764480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480"
Nov 02 18:00:04.689108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.689087    4476 generic.go:356] "Generic (PLEG): container finished" podID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerID="9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6" exitCode=60
Nov 02 18:00:04.689213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.689179    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerDied","Data":"9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6"}
Nov 02 18:00:04.689608 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.689591    4476 scope.go:122] "RemoveContainer" containerID="9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6"
Nov 02 18:00:04.689856 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:04.689833    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28_odh-ai-gateway-infra(b519242a-cca6-4bc3-afb8-fc718466eca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9"
Nov 02 18:00:04.691734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.691714    4476 generic.go:356] "Generic (PLEG): container finished" podID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerID="297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897" exitCode=60
Nov 02 18:00:04.691829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.691775    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerDied","Data":"297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897"}
Nov 02 18:00:04.692030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.692012    4476 scope.go:122] "RemoveContainer" containerID="297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897"
Nov 02 18:00:04.692249 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:04.692229    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w_odh-ai-gateway-infra(0d26c76a-495a-4fb7-9caa-5e05c8d3f912)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912"
Nov 02 18:00:04.698642 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.698608    4476 scope.go:122] "RemoveContainer" containerID="413c9285036bdbb6be4870e445cbf010968b8f08008fbe2b8e386a9e87ed5b39"
Nov 02 18:00:04.710202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.710181    4476 scope.go:122] "RemoveContainer" containerID="e57fefefc653b277410d7703ce0340b44a89107883217df7bfebb383fb790052"
Nov 02 18:00:04.721074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.721055    4476 scope.go:122] "RemoveContainer" containerID="1004f89c80b41e7125fae3c4b9bcd75fb7c646b71bbc21e2bd61e610db05f6f8"
Nov 02 18:00:04.731920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.731902    4476 scope.go:122] "RemoveContainer" containerID="2b918a5e6d1d9b81caa56c5fa50cce67024d28008f0866be27c02e0f16368485"
Nov 02 18:00:04.871675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:04.870010    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:05.377933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.377897    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 18:00:05.378125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.377954    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 18:00:05.382834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.382810    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 18:00:05.472653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.469178    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:05.700269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.700175    4476 scope.go:122] "RemoveContainer" containerID="3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa"
Nov 02 18:00:05.700771 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:05.700497    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx_odh-ai-gateway-infra(bf7d6fa8-3f58-4594-98f4-59eff2764480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480"
Nov 02 18:00:05.711595 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.711556    4476 scope.go:122] "RemoveContainer" containerID="9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6"
Nov 02 18:00:05.711882 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:05.711859    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28_odh-ai-gateway-infra(b519242a-cca6-4bc3-afb8-fc718466eca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9"
Nov 02 18:00:05.712882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.712865    4476 scope.go:122] "RemoveContainer" containerID="297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897"
Nov 02 18:00:05.713076 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:05.713054    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w_odh-ai-gateway-infra(0d26c76a-495a-4fb7-9caa-5e05c8d3f912)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912"
Nov 02 18:00:05.714516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.714501    4476 scope.go:122] "RemoveContainer" containerID="0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d"
Nov 02 18:00:05.714730 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:05.714707    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f_odh-ai-gateway-infra(23a9de3c-e58f-45f5-bca2-3db421ec9a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80"
Nov 02 18:00:05.716024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:05.716003    4476 scope.go:122] "RemoveContainer" containerID="d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b"
Nov 02 18:00:05.716223 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:05.716203    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z_odh-ai-gateway-infra(d086c8bf-5d20-48d4-9eec-5fcad95c40a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8"
Nov 02 18:00:06.377235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.377200    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 18:00:06.377421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.377251    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 18:00:06.479600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.479567    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"]
Nov 02 18:00:06.483065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.483048    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.522403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.522370    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttrd\" (UniqueName: \"kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.522643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.522599    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.536165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.536137    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d5924c-dockercfg-9dwhd\""
Nov 02 18:00:06.565711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.561249    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"]
Nov 02 18:00:06.596714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.596687    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d5924c\""
Nov 02 18:00:06.624201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.624171    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.624397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.624222    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttrd\" (UniqueName: \"kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.625067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.625039    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.682717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.682621    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttrd\" (UniqueName: \"kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:06.796163 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:06.796126    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:07.212370 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.212342    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"]
Nov 02 18:00:07.213776 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:07.213751    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b1e1dd_980e_48f9_8b33_9446f18dd355.slice/crio-0cbead8e8a79b6e1eb4669dbc37325d5b0adcbd3daaadfa09c312d9a72a54ec0 WatchSource:0}: Error finding container 0cbead8e8a79b6e1eb4669dbc37325d5b0adcbd3daaadfa09c312d9a72a54ec0: Status 404 returned error can't find the container with id 0cbead8e8a79b6e1eb4669dbc37325d5b0adcbd3daaadfa09c312d9a72a54ec0
Nov 02 18:00:07.287681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.287652    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"]
Nov 02 18:00:07.291477 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.291455    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.332496 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.332464    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.332667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.332507    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c54\" (UniqueName: \"kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54\") pod \"payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.384650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.383660    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 18:00:07.384650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.383716    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 18:00:07.394993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.394963    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"]
Nov 02 18:00:07.433976 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.433939    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.434146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.434003    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c54\" (UniqueName: \"kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54\") pod \"payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.434749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.434730    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.491090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.491014    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c54\" (UniqueName: \"kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54\") pod \"payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.605385 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.605355    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:07.732592 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.732392    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9" event={"ID":"79b1e1dd-980e-48f9-8b33-9446f18dd355","Type":"ContainerStarted","Data":"c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d"}
Nov 02 18:00:07.732592 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.732444    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9" event={"ID":"79b1e1dd-980e-48f9-8b33-9446f18dd355","Type":"ContainerStarted","Data":"0cbead8e8a79b6e1eb4669dbc37325d5b0adcbd3daaadfa09c312d9a72a54ec0"}
Nov 02 18:00:07.733235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.733215    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:07.873931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:07.873877    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9" podStartSLOduration=1.873860091 podStartE2EDuration="1.873860091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.851426768 +0000 UTC m=+2238.692032704" watchObservedRunningTime="2026-11-02 18:00:07.873860091 +0000 UTC m=+2238.714466024"
Nov 02 18:00:08.021907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.021866    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"]
Nov 02 18:00:08.029734 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:08.029696    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0469b71_13e4_48fc_a23d_0d0995410a9f.slice/crio-1efa2efc5eea495dfef3989bd5f06babdbccfe632b0db25dde2e531ed39a3969 WatchSource:0}: Error finding container 1efa2efc5eea495dfef3989bd5f06babdbccfe632b0db25dde2e531ed39a3969: Status 404 returned error can't find the container with id 1efa2efc5eea495dfef3989bd5f06babdbccfe632b0db25dde2e531ed39a3969
Nov 02 18:00:08.103023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.102158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:08.377209 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.377127    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 18:00:08.377209 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.377180    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 18:00:08.758704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.757985    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb" event={"ID":"c0469b71-13e4-48fc-a23d-0d0995410a9f","Type":"ContainerStarted","Data":"9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144"}
Nov 02 18:00:08.758704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.758041    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb" event={"ID":"c0469b71-13e4-48fc-a23d-0d0995410a9f","Type":"ContainerStarted","Data":"1efa2efc5eea495dfef3989bd5f06babdbccfe632b0db25dde2e531ed39a3969"}
Nov 02 18:00:08.758704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:08.758089    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:09.382061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:09.382026    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 18:00:09.382457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:09.382092    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 18:00:10.381911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:10.381883    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 18:00:10.382407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:10.382382    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 18:00:10.383222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:10.383201    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 18:00:10.553750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:10.553692    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb" podStartSLOduration=3.55367789 podStartE2EDuration="3.55367789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.884480147 +0000 UTC m=+2239.725086083" watchObservedRunningTime="2026-11-02 18:00:10.55367789 +0000 UTC m=+2241.394283855"
Nov 02 18:00:12.898225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:12.898191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:15.376804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:15.376772    4476 scope.go:122] "RemoveContainer" containerID="3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa"
Nov 02 18:00:15.377316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:15.376832    4476 scope.go:122] "RemoveContainer" containerID="d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b"
Nov 02 18:00:15.812181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:15.812143    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerStarted","Data":"587f7bb3a65170506e29cd96735cda1d2e4cb3852e63cd595922f21fc079bab2"}
Nov 02 18:00:15.814123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:15.814098    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerStarted","Data":"96d4b175127f4da84fec99fe2fdcd05a0da9b33e0a663377e81a93a6b12c2955"}
Nov 02 18:00:16.376578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.376541    4476 scope.go:122] "RemoveContainer" containerID="297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897"
Nov 02 18:00:16.823437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.823343    4476 generic.go:356] "Generic (PLEG): container finished" podID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerID="587f7bb3a65170506e29cd96735cda1d2e4cb3852e63cd595922f21fc079bab2" exitCode=60
Nov 02 18:00:16.823437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.823419    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerDied","Data":"587f7bb3a65170506e29cd96735cda1d2e4cb3852e63cd595922f21fc079bab2"}
Nov 02 18:00:16.824029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.823493    4476 scope.go:122] "RemoveContainer" containerID="d32cda9f9449cf7844fcec6791e8622e08cb0aced10d27c57ca13c98cd51d75b"
Nov 02 18:00:16.824029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.823909    4476 scope.go:122] "RemoveContainer" containerID="587f7bb3a65170506e29cd96735cda1d2e4cb3852e63cd595922f21fc079bab2"
Nov 02 18:00:16.824228 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:16.824207    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z_odh-ai-gateway-infra(d086c8bf-5d20-48d4-9eec-5fcad95c40a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8"
Nov 02 18:00:16.825717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.825695    4476 generic.go:356] "Generic (PLEG): container finished" podID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerID="96d4b175127f4da84fec99fe2fdcd05a0da9b33e0a663377e81a93a6b12c2955" exitCode=60
Nov 02 18:00:16.825810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.825764    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerDied","Data":"96d4b175127f4da84fec99fe2fdcd05a0da9b33e0a663377e81a93a6b12c2955"}
Nov 02 18:00:16.826103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.826089    4476 scope.go:122] "RemoveContainer" containerID="96d4b175127f4da84fec99fe2fdcd05a0da9b33e0a663377e81a93a6b12c2955"
Nov 02 18:00:16.826390 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:16.826371    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx_odh-ai-gateway-infra(bf7d6fa8-3f58-4594-98f4-59eff2764480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480"
Nov 02 18:00:16.827832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.827808    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerStarted","Data":"1933b1512b4c32836fc4547e70ad27e9ee26879d69fbf92bf11ad0686264069e"}
Nov 02 18:00:16.835885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.835867    4476 scope.go:122] "RemoveContainer" containerID="3745c40d1f4dae30fc2173025d5bf793d9143f872ec708b7e16f21facf87a1aa"
Nov 02 18:00:16.951453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:16.951420    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"]
Nov 02 18:00:17.074315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.074224    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"]
Nov 02 18:00:17.376658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.376412    4476 scope.go:122] "RemoveContainer" containerID="0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d"
Nov 02 18:00:17.753567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.753539    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:17.835889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.835860    4476 generic.go:356] "Generic (PLEG): container finished" podID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerID="1933b1512b4c32836fc4547e70ad27e9ee26879d69fbf92bf11ad0686264069e" exitCode=60
Nov 02 18:00:17.836349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.835941    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerDied","Data":"1933b1512b4c32836fc4547e70ad27e9ee26879d69fbf92bf11ad0686264069e"}
Nov 02 18:00:17.836349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.836012    4476 scope.go:122] "RemoveContainer" containerID="297555007c9236e74bfb3d706148ff41be978b3d444bbead6d6b40a555644897"
Nov 02 18:00:17.836487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.836366    4476 scope.go:122] "RemoveContainer" containerID="1933b1512b4c32836fc4547e70ad27e9ee26879d69fbf92bf11ad0686264069e"
Nov 02 18:00:17.836717 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:17.836677    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w_odh-ai-gateway-infra(0d26c76a-495a-4fb7-9caa-5e05c8d3f912)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912"
Nov 02 18:00:17.838148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:17.838126    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerStarted","Data":"851d9eab102f771e8c027e8b1775bbb7d582889c7c0de2b086b8eaaa878a7272"}
Nov 02 18:00:18.014755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.014729    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:18.067344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.067315    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:18.073898 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.073873    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:18.074098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.074079    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"]
Nov 02 18:00:18.093821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.093799    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9\") pod \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") "
Nov 02 18:00:18.094123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.094033    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf\" (UniqueName: \"kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf\") pod \"bf7d6fa8-3f58-4594-98f4-59eff2764480\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") "
Nov 02 18:00:18.094377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.094359    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca\") pod \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\" (UID: \"d086c8bf-5d20-48d4-9eec-5fcad95c40a8\") "
Nov 02 18:00:18.094466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.094387    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca\") pod \"bf7d6fa8-3f58-4594-98f4-59eff2764480\" (UID: \"bf7d6fa8-3f58-4594-98f4-59eff2764480\") "
Nov 02 18:00:18.094846 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.094786    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca" pod "d086c8bf-5d20-48d4-9eec-5fcad95c40a8" (UID: "d086c8bf-5d20-48d4-9eec-5fcad95c40a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:18.094982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.094876    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca" pod "bf7d6fa8-3f58-4594-98f4-59eff2764480" (UID: "bf7d6fa8-3f58-4594-98f4-59eff2764480"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:18.096326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.096297    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf" pod "bf7d6fa8-3f58-4594-98f4-59eff2764480" (UID: "bf7d6fa8-3f58-4594-98f4-59eff2764480"). InnerVolumeSpecName "kube-api-access-bqckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:18.096413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.096374    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9" pod "d086c8bf-5d20-48d4-9eec-5fcad95c40a8" (UID: "d086c8bf-5d20-48d4-9eec-5fcad95c40a8"). InnerVolumeSpecName "kube-api-access-jgfc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:18.195482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.195453    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqckf\" (UniqueName: \"kubernetes.io/projected/bf7d6fa8-3f58-4594-98f4-59eff2764480-kube-api-access-bqckf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:18.195482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.195480    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:18.195482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.195489    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d6fa8-3f58-4594-98f4-59eff2764480-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:18.195743 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.195498    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgfc9\" (UniqueName: \"kubernetes.io/projected/d086c8bf-5d20-48d4-9eec-5fcad95c40a8-kube-api-access-jgfc9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:18.375981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.375954    4476 scope.go:122] "RemoveContainer" containerID="9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6"
Nov 02 18:00:18.851677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.851646    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"
Nov 02 18:00:18.851677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.851668    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx" event={"ID":"bf7d6fa8-3f58-4594-98f4-59eff2764480","Type":"ContainerDied","Data":"b71da98555a85a9f9e88cabdf309902273fa8e6274dd8bad228a5313d2b13eb2"}
Nov 02 18:00:18.852228 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.851739    4476 scope.go:122] "RemoveContainer" containerID="96d4b175127f4da84fec99fe2fdcd05a0da9b33e0a663377e81a93a6b12c2955"
Nov 02 18:00:18.854083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.854056    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerStarted","Data":"2cb0e2706f58e5adc2b81a7059cff9d9d02c971d5b2bc8940ec1cf0c09d01172"}
Nov 02 18:00:18.858880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.858848    4476 generic.go:356] "Generic (PLEG): container finished" podID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerID="851d9eab102f771e8c027e8b1775bbb7d582889c7c0de2b086b8eaaa878a7272" exitCode=60
Nov 02 18:00:18.859003 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.858920    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerDied","Data":"851d9eab102f771e8c027e8b1775bbb7d582889c7c0de2b086b8eaaa878a7272"}
Nov 02 18:00:18.859326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.859307    4476 scope.go:122] "RemoveContainer" containerID="851d9eab102f771e8c027e8b1775bbb7d582889c7c0de2b086b8eaaa878a7272"
Nov 02 18:00:18.859656 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:18.859603    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f_odh-ai-gateway-infra(23a9de3c-e58f-45f5-bca2-3db421ec9a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80"
Nov 02 18:00:18.862288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.862267    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z" event={"ID":"d086c8bf-5d20-48d4-9eec-5fcad95c40a8","Type":"ContainerDied","Data":"e37d8467ebe2c85ed47c91b172732bbd1dbaf2ec62df9a98b57b469eaa0ec6a4"}
Nov 02 18:00:18.862377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.862348    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"
Nov 02 18:00:18.868936 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.868896    4476 scope.go:122] "RemoveContainer" containerID="0aec2a9aa88b6e76561b2930aac48ad521dc488f551e43480a36fe1cf6f4c93d"
Nov 02 18:00:18.889182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:18.889155    4476 scope.go:122] "RemoveContainer" containerID="587f7bb3a65170506e29cd96735cda1d2e4cb3852e63cd595922f21fc079bab2"
Nov 02 18:00:19.016156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.016125    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:19.020413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.020391    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"]
Nov 02 18:00:19.029751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.029726    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894040-m864z"]
Nov 02 18:00:19.106107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.106070    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca\") pod \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") "
Nov 02 18:00:19.106303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.106149    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb\" (UniqueName: \"kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb\") pod \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\" (UID: \"0d26c76a-495a-4fb7-9caa-5e05c8d3f912\") "
Nov 02 18:00:19.106502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.106478    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca" pod "0d26c76a-495a-4fb7-9caa-5e05c8d3f912" (UID: "0d26c76a-495a-4fb7-9caa-5e05c8d3f912"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:19.108353 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.108330    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb" pod "0d26c76a-495a-4fb7-9caa-5e05c8d3f912" (UID: "0d26c76a-495a-4fb7-9caa-5e05c8d3f912"). InnerVolumeSpecName "kube-api-access-wv9lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:19.207452 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.207372    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:19.207452 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.207405    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv9lb\" (UniqueName: \"kubernetes.io/projected/0d26c76a-495a-4fb7-9caa-5e05c8d3f912-kube-api-access-wv9lb\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:19.214789 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.214758    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"]
Nov 02 18:00:19.312964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.312938    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:19.366366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.366330    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"]
Nov 02 18:00:19.375875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.375844    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a099c3-29894040-vwlxx"]
Nov 02 18:00:19.385745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.385718    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" path="/var/lib/kubelet/pods/bf7d6fa8-3f58-4594-98f4-59eff2764480/volumes"
Nov 02 18:00:19.386117 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.386104    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" path="/var/lib/kubelet/pods/d086c8bf-5d20-48d4-9eec-5fcad95c40a8/volumes"
Nov 02 18:00:19.787650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.781808    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:00:19.919433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.919392    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w" event={"ID":"0d26c76a-495a-4fb7-9caa-5e05c8d3f912","Type":"ContainerDied","Data":"3c81d8d63137af27d940e3e58f1ade5513c53ed856c624e3d831a400cbc00eb9"}
Nov 02 18:00:19.921018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.920996    4476 scope.go:122] "RemoveContainer" containerID="1933b1512b4c32836fc4547e70ad27e9ee26879d69fbf92bf11ad0686264069e"
Nov 02 18:00:19.921325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.921311    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"
Nov 02 18:00:19.934271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.934227    4476 generic.go:356] "Generic (PLEG): container finished" podID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerID="2cb0e2706f58e5adc2b81a7059cff9d9d02c971d5b2bc8940ec1cf0c09d01172" exitCode=60
Nov 02 18:00:19.934403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.934296    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerDied","Data":"2cb0e2706f58e5adc2b81a7059cff9d9d02c971d5b2bc8940ec1cf0c09d01172"}
Nov 02 18:00:19.934771 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.934748    4476 scope.go:122] "RemoveContainer" containerID="2cb0e2706f58e5adc2b81a7059cff9d9d02c971d5b2bc8940ec1cf0c09d01172"
Nov 02 18:00:19.935103 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:19.935074    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28_odh-ai-gateway-infra(b519242a-cca6-4bc3-afb8-fc718466eca9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9"
Nov 02 18:00:19.940236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:19.940159    4476 scope.go:122] "RemoveContainer" containerID="9c40ad3089fe19c4e416f8ffcb983ab22193d917b22dd67bfa7c14c389853bf6"
Nov 02 18:00:20.135420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.135394    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:20.220963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.220934    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca\") pod \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") "
Nov 02 18:00:20.221175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.221049    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r\" (UniqueName: \"kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r\") pod \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\" (UID: \"23a9de3c-e58f-45f5-bca2-3db421ec9a80\") "
Nov 02 18:00:20.221340 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.221316    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca" pod "23a9de3c-e58f-45f5-bca2-3db421ec9a80" (UID: "23a9de3c-e58f-45f5-bca2-3db421ec9a80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.221441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.221426    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a9de3c-e58f-45f5-bca2-3db421ec9a80-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.223301 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.223272    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r" pod "23a9de3c-e58f-45f5-bca2-3db421ec9a80" (UID: "23a9de3c-e58f-45f5-bca2-3db421ec9a80"). InnerVolumeSpecName "kube-api-access-fr28r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.231055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.231025    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"]
Nov 02 18:00:20.243889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.243849    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894040-6cl6w"]
Nov 02 18:00:20.326269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.326234    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr28r\" (UniqueName: \"kubernetes.io/projected/23a9de3c-e58f-45f5-bca2-3db421ec9a80-kube-api-access-fr28r\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.504676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.502141    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"]
Nov 02 18:00:20.774543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.774451    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:00:20.956091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.956063    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"
Nov 02 18:00:20.956565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.956535    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f" event={"ID":"23a9de3c-e58f-45f5-bca2-3db421ec9a80","Type":"ContainerDied","Data":"e24476136b070c15a9b116db63e5ed4036e9eb48cdd1bd6be8ac2b25cc3f9ed1"}
Nov 02 18:00:20.956755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:20.956740    4476 scope.go:122] "RemoveContainer" containerID="851d9eab102f771e8c027e8b1775bbb7d582889c7c0de2b086b8eaaa878a7272"
Nov 02 18:00:21.144953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.144923    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:21.183002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.182963    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"]
Nov 02 18:00:21.188206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.188173    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f2289f-29894040-8cj2f"]
Nov 02 18:00:21.239559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.239517    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca\") pod \"b519242a-cca6-4bc3-afb8-fc718466eca9\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") "
Nov 02 18:00:21.239751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.239658    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp\" (UniqueName: \"kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp\") pod \"b519242a-cca6-4bc3-afb8-fc718466eca9\" (UID: \"b519242a-cca6-4bc3-afb8-fc718466eca9\") "
Nov 02 18:00:21.239982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.239954    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca" pod "b519242a-cca6-4bc3-afb8-fc718466eca9" (UID: "b519242a-cca6-4bc3-afb8-fc718466eca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:21.240107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.240089    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b519242a-cca6-4bc3-afb8-fc718466eca9-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.242450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.242416    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp" pod "b519242a-cca6-4bc3-afb8-fc718466eca9" (UID: "b519242a-cca6-4bc3-afb8-fc718466eca9"). InnerVolumeSpecName "kube-api-access-ptvmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:21.341583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.341495    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvmp\" (UniqueName: \"kubernetes.io/projected/b519242a-cca6-4bc3-afb8-fc718466eca9-kube-api-access-ptvmp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.384388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.384357    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" path="/var/lib/kubelet/pods/0d26c76a-495a-4fb7-9caa-5e05c8d3f912/volumes"
Nov 02 18:00:21.384844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.384829    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" path="/var/lib/kubelet/pods/23a9de3c-e58f-45f5-bca2-3db421ec9a80/volumes"
Nov 02 18:00:21.970838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.970801    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"
Nov 02 18:00:21.970838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.970818    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28" event={"ID":"b519242a-cca6-4bc3-afb8-fc718466eca9","Type":"ContainerDied","Data":"c708f8e19476d9cef91c232d8132498ad61857ac37e64a1a746129bad7c45b9a"}
Nov 02 18:00:21.971339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:21.970913    4476 scope.go:122] "RemoveContainer" containerID="2cb0e2706f58e5adc2b81a7059cff9d9d02c971d5b2bc8940ec1cf0c09d01172"
Nov 02 18:00:22.136097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:22.136060    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"]
Nov 02 18:00:22.184974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:22.184940    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ccf013-29894040-bft28"]
Nov 02 18:00:22.416183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:22.416150    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:22.618601 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:22.618573    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:23.383352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:23.383321    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" path="/var/lib/kubelet/pods/b519242a-cca6-4bc3-afb8-fc718466eca9/volumes"
Nov 02 18:00:27.362509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:27.362474    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:27.559765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:27.559732    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:27.749940 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:27.749819    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:29.059420 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:29.059377    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:00:32.111872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:32.111779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:32.744097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:32.744064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:37.859384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:37.859331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:38.576894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.576861    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"]
Nov 02 18:00:38.577443 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577427    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577445    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577454    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577460    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577473    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577480    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577490    4476 cpu_manager.go:414] "containerMap: removing container" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577496    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577506    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577514    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577522    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.577523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577527    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577536    4476 cpu_manager.go:414] "containerMap: removing container" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577540    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577546    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577551    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577561    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577568    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577573    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577578    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577588    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577592    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577601    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577605    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577702    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577710    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577715    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577721    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577727    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577734    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577742    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577749    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b519242a-cca6-4bc3-afb8-fc718466eca9" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577756    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577762    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577768    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577773    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:00:38.578110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.577781    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7d6fa8-3f58-4594-98f4-59eff2764480" containerName="cleanup"
Nov 02 18:00:38.582794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.582766    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.633423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.633394    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.633593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.633431    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.633593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.633510    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.633593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.633572    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.633717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.633696    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4kd\" (UniqueName: \"kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.633752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.633727    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.640024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.639987    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-943ca1\"/\"openshift-service-ca.crt\""
Nov 02 18:00:38.640173 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.640038    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-943ca1\"/\"kube-root-ca.crt\""
Nov 02 18:00:38.689737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.689695    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-943ca1\"/\"default-dockercfg-qtmwv\""
Nov 02 18:00:38.689933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.689808    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-943ca1\"/\"praxis-route-943ca1-kserve-self-signed-certs\""
Nov 02 18:00:38.693257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.693227    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"]
Nov 02 18:00:38.735580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.735538    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.735796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.735595    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.735796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.735681    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.735796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.735756    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.735963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.735814    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4kd\" (UniqueName: \"kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.735963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.735852    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.736067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.736047    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.736124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.736097    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.736192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.736169    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.738062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.738042    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.738429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.738407    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.793400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.793365    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4kd\" (UniqueName: \"kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd\") pod \"praxis-route-943ca1-kserve-755b5b8f55-nqrgb\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") " pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:38.902511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:38.902475    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:39.065823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:39.065567    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:39.306209 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:39.306177    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"]
Nov 02 18:00:39.307249 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:00:39.307224    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b7b2cce_a690_4133_8f81_add2738ca1f1.slice/crio-7d375851c4353b8b8b3e85ad74e106da1c1ad5cef2bc15a876ab16d411b50fa2 WatchSource:0}: Error finding container 7d375851c4353b8b8b3e85ad74e106da1c1ad5cef2bc15a876ab16d411b50fa2: Status 404 returned error can't find the container with id 7d375851c4353b8b8b3e85ad74e106da1c1ad5cef2bc15a876ab16d411b50fa2
Nov 02 18:00:40.112655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:40.112583    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb" event={"ID":"9b7b2cce-a690-4133-8f81-add2738ca1f1","Type":"ContainerStarted","Data":"ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf"}
Nov 02 18:00:40.112655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:40.112653    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb" event={"ID":"9b7b2cce-a690-4133-8f81-add2738ca1f1","Type":"ContainerStarted","Data":"7d375851c4353b8b8b3e85ad74e106da1c1ad5cef2bc15a876ab16d411b50fa2"}
Nov 02 18:00:40.232810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:40.232739    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb" podStartSLOduration=2.232713603 podStartE2EDuration="2.232713603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:40.229594127 +0000 UTC m=+2271.070200061" watchObservedRunningTime="2026-11-02 18:00:40.232713603 +0000 UTC m=+2271.073319541"
Nov 02 18:00:40.902803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:40.902766    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:42.460781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:42.460748    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:42.980675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:42.980621    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:47.857520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:47.857486    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:48.051958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:48.051897    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:48.903615 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:48.903577    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:48.916212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:48.916187    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:49.196825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:49.196734    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:00:49.355493 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:49.355452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:50.115716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:50.115682    4476 scope.go:122] "RemoveContainer" containerID="bbd3eaf7be97c379cabb217c89fc0333a88c682161b8449075d4554ca8efec16"
Nov 02 18:00:52.697991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:52.697952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:54.187026 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:54.186948    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:59.060274 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:00:59.060230    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:00:59.252863 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:59.252828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:59.474221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:00:59.474189    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:00.470560 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:00.470483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:01.074956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:01.074908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:01.724068 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:01.724032    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:04.194211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:04.194171    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:05.407877 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:05.407840    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:05.616866 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:05.616830    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:08.648653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:08.648585    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:08.839352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:08.839308    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:09.591051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:09.591018    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:14.552906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:14.552868    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:15.118088 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:15.118051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:18.065133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:18.065096    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:21.314303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:21.314245    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:23.778040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:23.778001    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:24.145558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:24.145526    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:24.424239 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:24.424155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:26.464226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:26.464195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:29.058234 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:01:29.058190    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:01:30.021326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:30.021296    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:31.636532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:31.636494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:36.364104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:36.364068    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:41.082762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:41.082723    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:45.992244 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:45.992211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:51.115443 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:51.115405    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:51.170521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:51.170476    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql_6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed/payload-processing/0.log"
Nov 02 18:01:51.320004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:51.319956    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:51.521553 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:51.521470    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:52.115429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:52.115403    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb_c0469b71-13e4-48fc-a23d-0d0995410a9f/payload-processing/0.log"
Nov 02 18:01:52.364139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:52.364106    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb_c0469b71-13e4-48fc-a23d-0d0995410a9f/payload-processing/0.log"
Nov 02 18:01:53.041509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:53.041464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:53.479685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:53.479647    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:55.941563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:55.941531    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:58.377657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:58.377609    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"]
Nov 02 18:01:58.495794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:01:58.495763    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:59.060659 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:01:59.060599    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:02:02.392421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:02.392385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:06.526472 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:06.526437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:08.288425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:08.288373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:10.895154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:10.895119    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:11.114177 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:11.114136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:12.460817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:12.460783    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:12.985492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:12.985451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:13.265512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.265410    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:13.398817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.398758    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh" podUID="0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" containerName="main" containerID="cri-o://d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5" gracePeriod=45
Nov 02 18:02:13.484135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.484104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:13.673014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.672994    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 18:02:13.717181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.717156    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:13.751731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751704    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir\") pod \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") "
Nov 02 18:02:13.751875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751749    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs\") pod \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") "
Nov 02 18:02:13.751875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751799    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2\" (UniqueName: \"kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2\") pod \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") "
Nov 02 18:02:13.751875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751831    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache\") pod \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") "
Nov 02 18:02:13.751875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751848    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home\") pod \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") "
Nov 02 18:02:13.752092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751948    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm\") pod \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\" (UID: \"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da\") "
Nov 02 18:02:13.752092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.751988    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir" pod "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" (UID: "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:13.752200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.752145    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache" pod "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" (UID: "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:13.752200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.752162    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home" pod "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" (UID: "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:13.752533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.752496    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:13.752533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.752522    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:13.752723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.752554    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:13.754685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.754657    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2" pod "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" (UID: "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da"). InnerVolumeSpecName "kube-api-access-t6ph2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:13.754823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.754711    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm" pod "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" (UID: "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:13.754823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.754799    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs" pod "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" (UID: "0b99fb76-c5c0-4fba-a62b-ae1d7e4104da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:02:13.812575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.812482    4476 generic.go:356] "Generic (PLEG): container finished" podID="0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" containerID="d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5" exitCode=0
Nov 02 18:02:13.812575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.812527    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"
Nov 02 18:02:13.812801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.812567    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh" event={"ID":"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da","Type":"ContainerDied","Data":"d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5"}
Nov 02 18:02:13.812801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.812617    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh" event={"ID":"0b99fb76-c5c0-4fba-a62b-ae1d7e4104da","Type":"ContainerDied","Data":"04c0f24ebae8c8bc0d5d2a22ff954e3863dc971708b6931ddd960908b3898b65"}
Nov 02 18:02:13.812801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.812659    4476 scope.go:122] "RemoveContainer" containerID="d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5"
Nov 02 18:02:13.826102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.826082    4476 scope.go:122] "RemoveContainer" containerID="d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5"
Nov 02 18:02:13.826455 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:02:13.826430    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5\": container with ID starting with d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5 not found: ID does not exist" containerID="d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5"
Nov 02 18:02:13.826555 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.826465    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5"} err="failed to get container status \"d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5\": rpc error: code = NotFound desc = could not find container \"d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5\": container with ID starting with d269357c7f4edb2d97975ac57ff031be2374beaae32f685c7cadf837c3f196e5 not found: ID does not exist"
Nov 02 18:02:13.854202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.854177    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:13.854202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.854202    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ph2\" (UniqueName: \"kubernetes.io/projected/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-kube-api-access-t6ph2\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:13.854402 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.854213    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:13.978510 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.978480    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"]
Nov 02 18:02:13.981816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:13.981793    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ccf013/ipp-route-ccf013-kserve-5974b5b445-j45wh"]
Nov 02 18:02:15.387882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:15.387843    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" path="/var/lib/kubelet/pods/0b99fb76-c5c0-4fba-a62b-ae1d7e4104da/volumes"
Nov 02 18:02:18.586585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:18.586515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:19.832967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:19.832934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:20.037115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:20.037044    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:20.417884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:20.417853    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"]
Nov 02 18:02:20.834817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:20.834718    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:21.039493 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:21.039462    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:23.214201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:23.214166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:25.295573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.295543    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:25.418513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.418477    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s" podUID="2905fa28-45e0-4145-9d16-d008e84ce6da" containerName="payload-processing" containerID="cri-o://b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3" gracePeriod=25
Nov 02 18:02:25.679837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.679770    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:25.697997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.697973    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 18:02:25.803663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.802072    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume\") pod \"2905fa28-45e0-4145-9d16-d008e84ce6da\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") "
Nov 02 18:02:25.803663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.802289    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z\" (UniqueName: \"kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z\") pod \"2905fa28-45e0-4145-9d16-d008e84ce6da\" (UID: \"2905fa28-45e0-4145-9d16-d008e84ce6da\") "
Nov 02 18:02:25.805119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.805069    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume" pod "2905fa28-45e0-4145-9d16-d008e84ce6da" (UID: "2905fa28-45e0-4145-9d16-d008e84ce6da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:25.811133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.811097    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z" pod "2905fa28-45e0-4145-9d16-d008e84ce6da" (UID: "2905fa28-45e0-4145-9d16-d008e84ce6da"). InnerVolumeSpecName "kube-api-access-wfh7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:25.877679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.877616    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:25.904094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.904063    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfh7z\" (UniqueName: \"kubernetes.io/projected/2905fa28-45e0-4145-9d16-d008e84ce6da-kube-api-access-wfh7z\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:25.904094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.904090    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2905fa28-45e0-4145-9d16-d008e84ce6da-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:25.911774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.911747    4476 generic.go:356] "Generic (PLEG): container finished" podID="2905fa28-45e0-4145-9d16-d008e84ce6da" containerID="b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3" exitCode=0
Nov 02 18:02:25.911905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.911811    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"
Nov 02 18:02:25.911905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.911836    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s" event={"ID":"2905fa28-45e0-4145-9d16-d008e84ce6da","Type":"ContainerDied","Data":"b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3"}
Nov 02 18:02:25.911995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.911919    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s" event={"ID":"2905fa28-45e0-4145-9d16-d008e84ce6da","Type":"ContainerDied","Data":"893d582fbe0ad75cd3e8175153a5a3e4db6139084a0ae702ac7adcc55bc10b8b"}
Nov 02 18:02:25.911995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.911957    4476 scope.go:122] "RemoveContainer" containerID="b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3"
Nov 02 18:02:25.923514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.923489    4476 scope.go:122] "RemoveContainer" containerID="b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3"
Nov 02 18:02:25.923885 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:02:25.923863    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3\": container with ID starting with b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3 not found: ID does not exist" containerID="b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3"
Nov 02 18:02:25.923975 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:25.923895    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3"} err="failed to get container status \"b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3\": rpc error: code = NotFound desc = could not find container \"b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3\": container with ID starting with b68df78a98b2fab19e5951124e9cc93964f448a7e757ab073f122fe795dbe3b3 not found: ID does not exist"
Nov 02 18:02:26.084955 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:26.084921    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"]
Nov 02 18:02:26.088194 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:26.088171    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f42381-68d8db9cb5-c6w9s"]
Nov 02 18:02:27.320768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:27.320731    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:27.384927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:27.384883    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2905fa28-45e0-4145-9d16-d008e84ce6da" path="/var/lib/kubelet/pods/2905fa28-45e0-4145-9d16-d008e84ce6da/volumes"
Nov 02 18:02:27.817062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:27.817031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:29.060794 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:02:29.060760    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:02:32.080573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:32.080452    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:34.720849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:34.720820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:35.194906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.194873    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"]
Nov 02 18:02:35.195384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195370    4476 cpu_manager.go:414] "containerMap: removing container" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195384    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195406    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195411    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d086c8bf-5d20-48d4-9eec-5fcad95c40a8" containerName="cleanup"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195422    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195427    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195448    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2905fa28-45e0-4145-9d16-d008e84ce6da" containerName="payload-processing"
Nov 02 18:02:35.195454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195452    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2905fa28-45e0-4145-9d16-d008e84ce6da" containerName="payload-processing"
Nov 02 18:02:35.195822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195460    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" containerName="main"
Nov 02 18:02:35.195822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195465    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" containerName="main"
Nov 02 18:02:35.195822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195546    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26c76a-495a-4fb7-9caa-5e05c8d3f912" containerName="cleanup"
Nov 02 18:02:35.195822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195556    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2905fa28-45e0-4145-9d16-d008e84ce6da" containerName="payload-processing"
Nov 02 18:02:35.195822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195563    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b99fb76-c5c0-4fba-a62b-ae1d7e4104da" containerName="main"
Nov 02 18:02:35.195822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.195570    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a9de3c-e58f-45f5-bca2-3db421ec9a80" containerName="cleanup"
Nov 02 18:02:35.200060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.200040    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.257329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.257295    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"]
Nov 02 18:02:35.300735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.300701    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:02:35.300917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.300841    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:02:35.321906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.318337    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jqj7\" (UniqueName: \"kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.321906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.318762    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.321906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.318938    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.419809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.419776    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.420012 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.419881    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jqj7\" (UniqueName: \"kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.420012 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.419945    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.420554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.420524    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.422499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.422461    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.481445 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.481366    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jqj7\" (UniqueName: \"kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7\") pod \"maas-api-revoke-sub-cc07ab0f-8p7vs\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.514065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.514036    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:35.895762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:35.895739    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"]
Nov 02 18:02:35.897774 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:02:35.897745    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5d28d1_30e7_4254_a850_bf6aafb4df25.slice/crio-13b771bdd3c6e31cb106e44059001819393612a7aa0901416549c0cfe8340e21 WatchSource:0}: Error finding container 13b771bdd3c6e31cb106e44059001819393612a7aa0901416549c0cfe8340e21: Status 404 returned error can't find the container with id 13b771bdd3c6e31cb106e44059001819393612a7aa0901416549c0cfe8340e21
Nov 02 18:02:36.002024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:36.001987    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs" event={"ID":"1d5d28d1-30e7-4254-a850-bf6aafb4df25","Type":"ContainerStarted","Data":"13b771bdd3c6e31cb106e44059001819393612a7aa0901416549c0cfe8340e21"}
Nov 02 18:02:36.034920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:36.034888    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:37.010566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:37.010483    4476 generic.go:356] "Generic (PLEG): container finished" podID="1d5d28d1-30e7-4254-a850-bf6aafb4df25" containerID="d8e0f5f29b1b2e2523388af1876bee92bf46a0888c21b53fb13a9c68329bd951" exitCode=0
Nov 02 18:02:37.010566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:37.010517    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs" event={"ID":"1d5d28d1-30e7-4254-a850-bf6aafb4df25","Type":"ContainerDied","Data":"d8e0f5f29b1b2e2523388af1876bee92bf46a0888c21b53fb13a9c68329bd951"}
Nov 02 18:02:37.091669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:37.091621    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:38.184419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.184395    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:38.250676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.250641    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca\") pod \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") "
Nov 02 18:02:38.250841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.250818    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7\" (UniqueName: \"kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7\") pod \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") "
Nov 02 18:02:38.250933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.250865    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca\") pod \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\" (UID: \"1d5d28d1-30e7-4254-a850-bf6aafb4df25\") "
Nov 02 18:02:38.251041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.251002    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca" pod "1d5d28d1-30e7-4254-a850-bf6aafb4df25" (UID: "1d5d28d1-30e7-4254-a850-bf6aafb4df25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:38.251342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.251316    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:38.253692 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.253670    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca" pod "1d5d28d1-30e7-4254-a850-bf6aafb4df25" (UID: "1d5d28d1-30e7-4254-a850-bf6aafb4df25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:02:38.253813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.253714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7" pod "1d5d28d1-30e7-4254-a850-bf6aafb4df25" (UID: "1d5d28d1-30e7-4254-a850-bf6aafb4df25"). InnerVolumeSpecName "kube-api-access-8jqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:38.353002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.352917    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jqj7\" (UniqueName: \"kubernetes.io/projected/1d5d28d1-30e7-4254-a850-bf6aafb4df25-kube-api-access-8jqj7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:38.353002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:38.352955    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d5d28d1-30e7-4254-a850-bf6aafb4df25-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:39.024090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:39.024061    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"
Nov 02 18:02:39.024270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:39.024107    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs" event={"ID":"1d5d28d1-30e7-4254-a850-bf6aafb4df25","Type":"ContainerDied","Data":"13b771bdd3c6e31cb106e44059001819393612a7aa0901416549c0cfe8340e21"}
Nov 02 18:02:39.024270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:39.024161    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b771bdd3c6e31cb106e44059001819393612a7aa0901416549c0cfe8340e21"
Nov 02 18:02:42.676222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.676185    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"]
Nov 02 18:02:42.676734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.676720    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1d5d28d1-30e7-4254-a850-bf6aafb4df25" containerName="revoke-keys"
Nov 02 18:02:42.676787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.676736    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5d28d1-30e7-4254-a850-bf6aafb4df25" containerName="revoke-keys"
Nov 02 18:02:42.676863 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.676854    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5d28d1-30e7-4254-a850-bf6aafb4df25" containerName="revoke-keys"
Nov 02 18:02:42.680443 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.680422    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.733116 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.733089    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:02:42.738412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.738386    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"]
Nov 02 18:02:42.782778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.782744    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:02:42.805329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.805295    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmlt\" (UniqueName: \"kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.805512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.805358    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.805589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.805519    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.877334 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.877296    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:42.907151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.907118    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmlt\" (UniqueName: \"kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.907305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.907185    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.907305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.907221    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.907881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.907855    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.909741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.909722    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.964103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.964036    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmlt\" (UniqueName: \"kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt\") pod \"maas-api-revoke-sub-e0d2809f-zwp8q\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:42.994182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:42.994155    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:43.076858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:43.076801    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:43.375225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:43.375198    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"]
Nov 02 18:02:43.376180 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:02:43.376154    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33668b28_3708_425e_80b2_06f95a98305c.slice/crio-fd25df02a1b32eb3f82ca0ddbd79951b68c4cec8def5f0bf040c7fb0495eae8c WatchSource:0}: Error finding container fd25df02a1b32eb3f82ca0ddbd79951b68c4cec8def5f0bf040c7fb0495eae8c: Status 404 returned error can't find the container with id fd25df02a1b32eb3f82ca0ddbd79951b68c4cec8def5f0bf040c7fb0495eae8c
Nov 02 18:02:44.064689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:44.064576    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q" event={"ID":"33668b28-3708-425e-80b2-06f95a98305c","Type":"ContainerStarted","Data":"bfe3dcec413e2bab8dc44d922bacc34f9132a603fe3e09ed0dc562f9919dee31"}
Nov 02 18:02:44.064689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:44.064668    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q" event={"ID":"33668b28-3708-425e-80b2-06f95a98305c","Type":"ContainerStarted","Data":"fd25df02a1b32eb3f82ca0ddbd79951b68c4cec8def5f0bf040c7fb0495eae8c"}
Nov 02 18:02:44.185725 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:44.185676    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q" podStartSLOduration=2.185662057 podStartE2EDuration="2.185662057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:44.18453076 +0000 UTC m=+2395.025136696" watchObservedRunningTime="2026-11-02 18:02:44.185662057 +0000 UTC m=+2395.026267987"
Nov 02 18:02:45.072621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:45.072585    4476 generic.go:356] "Generic (PLEG): container finished" podID="33668b28-3708-425e-80b2-06f95a98305c" containerID="bfe3dcec413e2bab8dc44d922bacc34f9132a603fe3e09ed0dc562f9919dee31" exitCode=0
Nov 02 18:02:45.073065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:45.072662    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q" event={"ID":"33668b28-3708-425e-80b2-06f95a98305c","Type":"ContainerDied","Data":"bfe3dcec413e2bab8dc44d922bacc34f9132a603fe3e09ed0dc562f9919dee31"}
Nov 02 18:02:46.240733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.240704    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:46.348939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.348860    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt\" (UniqueName: \"kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt\") pod \"33668b28-3708-425e-80b2-06f95a98305c\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") "
Nov 02 18:02:46.349114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.348954    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca\") pod \"33668b28-3708-425e-80b2-06f95a98305c\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") "
Nov 02 18:02:46.349114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.349007    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca\") pod \"33668b28-3708-425e-80b2-06f95a98305c\" (UID: \"33668b28-3708-425e-80b2-06f95a98305c\") "
Nov 02 18:02:46.349418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.349390    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca" pod "33668b28-3708-425e-80b2-06f95a98305c" (UID: "33668b28-3708-425e-80b2-06f95a98305c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:46.351272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.351237    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt" pod "33668b28-3708-425e-80b2-06f95a98305c" (UID: "33668b28-3708-425e-80b2-06f95a98305c"). InnerVolumeSpecName "kube-api-access-7gmlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:46.351393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.351297    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca" pod "33668b28-3708-425e-80b2-06f95a98305c" (UID: "33668b28-3708-425e-80b2-06f95a98305c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:02:46.450084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.450057    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmlt\" (UniqueName: \"kubernetes.io/projected/33668b28-3708-425e-80b2-06f95a98305c-kube-api-access-7gmlt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:46.450084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.450083    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33668b28-3708-425e-80b2-06f95a98305c-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:46.450084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.450092    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33668b28-3708-425e-80b2-06f95a98305c-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:02:46.795879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:46.795827    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:47.090847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:47.090757    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"
Nov 02 18:02:47.090847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:47.090768    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q" event={"ID":"33668b28-3708-425e-80b2-06f95a98305c","Type":"ContainerDied","Data":"fd25df02a1b32eb3f82ca0ddbd79951b68c4cec8def5f0bf040c7fb0495eae8c"}
Nov 02 18:02:47.090847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:47.090836    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd25df02a1b32eb3f82ca0ddbd79951b68c4cec8def5f0bf040c7fb0495eae8c"
Nov 02 18:02:47.820047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:47.820009    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:49.748785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:49.748757    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:02:49.757543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:49.757523    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:02:51.741247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:51.741217    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:53.399653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:53.399608    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:54.761030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:54.760994    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"]
Nov 02 18:02:54.761474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:54.761234    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy" containerID="cri-o://d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03" gracePeriod=30
Nov 02 18:02:55.257691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:55.257607    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:55.738437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:55.738397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:58.293118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:02:58.293073    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:59.065025 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:02:59.064981    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:03:00.019442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.019417    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 18:03:00.117895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.117868    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.117978    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118014    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118036    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118080    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118122    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118154    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118187    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118233    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs\") pod \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\" (UID: \"4e6ad33a-a039-4e1d-acc4-5b6a99583279\") "
Nov 02 18:03:00.118407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118342    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:00.118515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118459    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:00.118680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118622    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:00.118769 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118680    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:00.118823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118812    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.118894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118829    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.118894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118841    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.118894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118854    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.119214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.118936    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:00.120298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.120270    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:00.120511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.120488    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "kube-api-access-5f4wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:00.120576 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.120558    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:00.120805 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.120789    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token" pod "4e6ad33a-a039-4e1d-acc4-5b6a99583279" (UID: "4e6ad33a-a039-4e1d-acc4-5b6a99583279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:00.181977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.181944    4476 generic.go:356] "Generic (PLEG): container finished" podID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerID="d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03" exitCode=0
Nov 02 18:03:00.182187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.182018    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"
Nov 02 18:03:00.182187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.182095    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" event={"ID":"4e6ad33a-a039-4e1d-acc4-5b6a99583279","Type":"ContainerDied","Data":"d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03"}
Nov 02 18:03:00.182187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.182141    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk" event={"ID":"4e6ad33a-a039-4e1d-acc4-5b6a99583279","Type":"ContainerDied","Data":"eaf4878109f583140642c09e5a717c7a8f63d7ac1b8a860206a0bf7ba7ead447"}
Nov 02 18:03:00.182187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.182171    4476 scope.go:122] "RemoveContainer" containerID="d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03"
Nov 02 18:03:00.195429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.195410    4476 scope.go:122] "RemoveContainer" containerID="d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03"
Nov 02 18:03:00.195787 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:00.195765    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03\": container with ID starting with d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03 not found: ID does not exist" containerID="d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03"
Nov 02 18:03:00.195870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.195795    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03"} err="failed to get container status \"d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03\": rpc error: code = NotFound desc = could not find container \"d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03\": container with ID starting with d29cf4c1783a15cfde394cdb072bb72f8c5f293626efcbdcf2f7e7e0977a6e03 not found: ID does not exist"
Nov 02 18:03:00.220171 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.220136    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4wt\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-kube-api-access-5f4wt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.220171 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.220164    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.220328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.220181    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.220328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.220193    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.220328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.220207    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6ad33a-a039-4e1d-acc4-5b6a99583279-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:00.360358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.360206    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"]
Nov 02 18:03:00.365099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.365066    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-spgxk"]
Nov 02 18:03:00.704378 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.704337    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"]
Nov 02 18:03:00.704916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.704900    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy"
Nov 02 18:03:00.704916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.704917    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy"
Nov 02 18:03:00.705060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.704927    4476 cpu_manager.go:414] "containerMap: removing container" podUID="33668b28-3708-425e-80b2-06f95a98305c" containerName="revoke-keys"
Nov 02 18:03:00.705060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.704932    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33668b28-3708-425e-80b2-06f95a98305c" containerName="revoke-keys"
Nov 02 18:03:00.705060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.705004    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" containerName="istio-proxy"
Nov 02 18:03:00.705060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.705012    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="33668b28-3708-425e-80b2-06f95a98305c" containerName="revoke-keys"
Nov 02 18:03:00.709604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.709582    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.762961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.762927    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:03:00.763615 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.763593    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:03:00.765080 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.765056    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"]
Nov 02 18:03:00.826488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.826455    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8vc\" (UniqueName: \"kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.826488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.826493    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.826723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.826676    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.927375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.927338    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8vc\" (UniqueName: \"kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.927564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.927382    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.927564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.927472    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.928186 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.928160    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.930078 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.930056    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:00.986878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:00.986811    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8vc\" (UniqueName: \"kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc\") pod \"maas-api-revoke-sub-9c7c3511-tb9bk\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:01.023467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:01.023434    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:01.384394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:01.384362    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6ad33a-a039-4e1d-acc4-5b6a99583279" path="/var/lib/kubelet/pods/4e6ad33a-a039-4e1d-acc4-5b6a99583279/volumes"
Nov 02 18:03:01.424244 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:01.424216    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"]
Nov 02 18:03:01.425222 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:03:01.425195    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65aa76e_6384_45f9_8b7a_97a919742be6.slice/crio-e950ad6f1f036d48787f5de47e776f1deed3eed9bc1ae05708ad49d4b356bac3 WatchSource:0}: Error finding container e950ad6f1f036d48787f5de47e776f1deed3eed9bc1ae05708ad49d4b356bac3: Status 404 returned error can't find the container with id e950ad6f1f036d48787f5de47e776f1deed3eed9bc1ae05708ad49d4b356bac3
Nov 02 18:03:02.207447 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:02.207404    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk" event={"ID":"b65aa76e-6384-45f9-8b7a-97a919742be6","Type":"ContainerStarted","Data":"a4f4f683549ac15a650d312b1a7fb8ea1ccecf9bfac4b93be2a7d5e5f71e55d1"}
Nov 02 18:03:02.207936 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:02.207477    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk" event={"ID":"b65aa76e-6384-45f9-8b7a-97a919742be6","Type":"ContainerStarted","Data":"e950ad6f1f036d48787f5de47e776f1deed3eed9bc1ae05708ad49d4b356bac3"}
Nov 02 18:03:02.330187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:02.330133    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk" podStartSLOduration=2.330119306 podStartE2EDuration="2.330119306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:02.327792445 +0000 UTC m=+2413.168398382" watchObservedRunningTime="2026-11-02 18:03:02.330119306 +0000 UTC m=+2413.170725239"
Nov 02 18:03:03.088220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:03.088187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:03.215746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:03.215711    4476 generic.go:356] "Generic (PLEG): container finished" podID="b65aa76e-6384-45f9-8b7a-97a919742be6" containerID="a4f4f683549ac15a650d312b1a7fb8ea1ccecf9bfac4b93be2a7d5e5f71e55d1" exitCode=0
Nov 02 18:03:03.216161 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:03.215792    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk" event={"ID":"b65aa76e-6384-45f9-8b7a-97a919742be6","Type":"ContainerDied","Data":"a4f4f683549ac15a650d312b1a7fb8ea1ccecf9bfac4b93be2a7d5e5f71e55d1"}
Nov 02 18:03:04.408808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.408777    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:04.579401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.579306    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca\") pod \"b65aa76e-6384-45f9-8b7a-97a919742be6\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") "
Nov 02 18:03:04.579401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.579388    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc\" (UniqueName: \"kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc\") pod \"b65aa76e-6384-45f9-8b7a-97a919742be6\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") "
Nov 02 18:03:04.579657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.579457    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca\") pod \"b65aa76e-6384-45f9-8b7a-97a919742be6\" (UID: \"b65aa76e-6384-45f9-8b7a-97a919742be6\") "
Nov 02 18:03:04.579883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.579853    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca" pod "b65aa76e-6384-45f9-8b7a-97a919742be6" (UID: "b65aa76e-6384-45f9-8b7a-97a919742be6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:04.581769 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.581738    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca" pod "b65aa76e-6384-45f9-8b7a-97a919742be6" (UID: "b65aa76e-6384-45f9-8b7a-97a919742be6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:04.581855 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.581774    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc" pod "b65aa76e-6384-45f9-8b7a-97a919742be6" (UID: "b65aa76e-6384-45f9-8b7a-97a919742be6"). InnerVolumeSpecName "kube-api-access-nh8vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:04.681307 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.681267    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:04.681307 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.681301    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b65aa76e-6384-45f9-8b7a-97a919742be6-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:04.681307 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:04.681311    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8vc\" (UniqueName: \"kubernetes.io/projected/b65aa76e-6384-45f9-8b7a-97a919742be6-kube-api-access-nh8vc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:05.238316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:05.238286    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"
Nov 02 18:03:05.238508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:05.238288    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk" event={"ID":"b65aa76e-6384-45f9-8b7a-97a919742be6","Type":"ContainerDied","Data":"e950ad6f1f036d48787f5de47e776f1deed3eed9bc1ae05708ad49d4b356bac3"}
Nov 02 18:03:05.238508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:05.238415    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e950ad6f1f036d48787f5de47e776f1deed3eed9bc1ae05708ad49d4b356bac3"
Nov 02 18:03:05.868926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:05.868893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:07.295593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.295554    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:07.628200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.628163    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"]
Nov 02 18:03:07.628416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.628387    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql" podUID="6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" containerName="payload-processing" containerID="cri-o://c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377" gracePeriod=30
Nov 02 18:03:07.684213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.684168    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"]
Nov 02 18:03:07.684434 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.684374    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8" podUID="2daa75d4-fa02-48e1-8981-e9704a3552ed" containerName="payload-pre-processing" containerID="cri-o://fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff" gracePeriod=30
Nov 02 18:03:07.886206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.885067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:07.975178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:07.975152    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 18:03:08.016388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.016366    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 18:03:08.078055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.078021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:08.119843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.119702    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z\" (UniqueName: \"kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z\") pod \"2daa75d4-fa02-48e1-8981-e9704a3552ed\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") "
Nov 02 18:03:08.119843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.119795    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm\" (UniqueName: \"kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm\") pod \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") "
Nov 02 18:03:08.120256 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.120182    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume\") pod \"2daa75d4-fa02-48e1-8981-e9704a3552ed\" (UID: \"2daa75d4-fa02-48e1-8981-e9704a3552ed\") "
Nov 02 18:03:08.120403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.120346    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume\") pod \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\" (UID: \"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed\") "
Nov 02 18:03:08.120533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.120514    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume" pod "2daa75d4-fa02-48e1-8981-e9704a3552ed" (UID: "2daa75d4-fa02-48e1-8981-e9704a3552ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:08.120808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.120775    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume" pod "6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" (UID: "6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:08.120899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.120868    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daa75d4-fa02-48e1-8981-e9704a3552ed-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:08.122573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.122555    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z" pod "2daa75d4-fa02-48e1-8981-e9704a3552ed" (UID: "2daa75d4-fa02-48e1-8981-e9704a3552ed"). InnerVolumeSpecName "kube-api-access-dwr2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:08.122676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.122621    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm" pod "6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" (UID: "6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed"). InnerVolumeSpecName "kube-api-access-z79fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:08.222330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.222248    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79fm\" (UniqueName: \"kubernetes.io/projected/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-kube-api-access-z79fm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:08.222330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.222288    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:08.222330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.222318    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwr2z\" (UniqueName: \"kubernetes.io/projected/2daa75d4-fa02-48e1-8981-e9704a3552ed-kube-api-access-dwr2z\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:08.264584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.264546    4476 generic.go:356] "Generic (PLEG): container finished" podID="2daa75d4-fa02-48e1-8981-e9704a3552ed" containerID="fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff" exitCode=0
Nov 02 18:03:08.264794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.264607    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"
Nov 02 18:03:08.264794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.264669    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8" event={"ID":"2daa75d4-fa02-48e1-8981-e9704a3552ed","Type":"ContainerDied","Data":"fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff"}
Nov 02 18:03:08.264794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.264727    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8" event={"ID":"2daa75d4-fa02-48e1-8981-e9704a3552ed","Type":"ContainerDied","Data":"ec7cbde76c35c9728f96f1a586e452e3a97f891057a7c3d2b3ba82a136257821"}
Nov 02 18:03:08.264794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.264749    4476 scope.go:122] "RemoveContainer" containerID="fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff"
Nov 02 18:03:08.266403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.266379    4476 generic.go:356] "Generic (PLEG): container finished" podID="6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" containerID="c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377" exitCode=0
Nov 02 18:03:08.266494 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.266468    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"
Nov 02 18:03:08.266554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.266531    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql" event={"ID":"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed","Type":"ContainerDied","Data":"c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377"}
Nov 02 18:03:08.266606 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.266578    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql" event={"ID":"6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed","Type":"ContainerDied","Data":"9793aaa1583ec6f8b4816985defea0e3b78cbc4d0994dc04056730ecd4d1b5ab"}
Nov 02 18:03:08.278671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.278621    4476 scope.go:122] "RemoveContainer" containerID="fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff"
Nov 02 18:03:08.278965 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:08.278943    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff\": container with ID starting with fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff not found: ID does not exist" containerID="fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff"
Nov 02 18:03:08.279026 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.278976    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff"} err="failed to get container status \"fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff\": rpc error: code = NotFound desc = could not find container \"fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff\": container with ID starting with fd21abf192a9bd390e3d0b38962e2b3febdaee5c48c41b2c2286a82fb5c75aff not found: ID does not exist"
Nov 02 18:03:08.279026 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.278997    4476 scope.go:122] "RemoveContainer" containerID="c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377"
Nov 02 18:03:08.288850 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.288834    4476 scope.go:122] "RemoveContainer" containerID="c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377"
Nov 02 18:03:08.289119 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:08.289098    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377\": container with ID starting with c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377 not found: ID does not exist" containerID="c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377"
Nov 02 18:03:08.289168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.289127    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377"} err="failed to get container status \"c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377\": rpc error: code = NotFound desc = could not find container \"c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377\": container with ID starting with c598a78a75bf9f211ed6703f3e49315906bb949e0ca66912a9cc8abc0763a377 not found: ID does not exist"
Nov 02 18:03:08.429368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.429335    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"]
Nov 02 18:03:08.433867 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.433844    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ccf013-85ff4f5567-92cr8"]
Nov 02 18:03:08.545800 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.545714    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"]
Nov 02 18:03:08.550657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:08.550619    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ccf013-7495d98986-tttql"]
Nov 02 18:03:09.156889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:09.156811    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:09.384300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:09.384267    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2daa75d4-fa02-48e1-8981-e9704a3552ed" path="/var/lib/kubelet/pods/2daa75d4-fa02-48e1-8981-e9704a3552ed/volumes"
Nov 02 18:03:09.384714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:09.384699    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" path="/var/lib/kubelet/pods/6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed/volumes"
Nov 02 18:03:10.779175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:10.779138    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"]
Nov 02 18:03:10.897379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:10.897343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:11.260166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:11.260120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:11.461660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:11.461601    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:12.790451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:12.790420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:13.234899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:13.234869    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"]
Nov 02 18:03:13.235095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:13.235075    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy" containerID="cri-o://0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd" gracePeriod=30
Nov 02 18:03:13.283248 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:13.283208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:14.048292 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:14.048252    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:14.612132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:14.612098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:15.229994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:15.229957    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:15.467017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:15.466983    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:18.521176 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.521150    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 18:03:18.636566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636533    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636592    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636748    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636797    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636834    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636867    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636890    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636908    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.636980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.636930    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs\") pod \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\" (UID: \"ab0c4a68-ea6a-43ea-958a-87be9d86be8f\") "
Nov 02 18:03:18.637436 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.637335    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:18.637621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.637594    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.637776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.637701    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:18.637931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.637902    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:18.638269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.638216    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:18.638879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.638847    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:18.639722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.639687    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "kube-api-access-zhsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:18.640492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.640459    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:18.641038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.641016    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:18.641455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.641432    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy" pod "ab0c4a68-ea6a-43ea-958a-87be9d86be8f" (UID: "ab0c4a68-ea6a-43ea-958a-87be9d86be8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:18.738691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738661    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738688    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738698    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738709    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsv4\" (UniqueName: \"kubernetes.io/projected/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-kube-api-access-zhsv4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738720    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738728    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738737    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:18.738952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:18.738745    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c4a68-ea6a-43ea-958a-87be9d86be8f-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:19.003321 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.003284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:19.351146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.351074    4476 generic.go:356] "Generic (PLEG): container finished" podID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerID="0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd" exitCode=0
Nov 02 18:03:19.351290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.351166    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"
Nov 02 18:03:19.351290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.351165    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" event={"ID":"ab0c4a68-ea6a-43ea-958a-87be9d86be8f","Type":"ContainerDied","Data":"0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd"}
Nov 02 18:03:19.351290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.351243    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9" event={"ID":"ab0c4a68-ea6a-43ea-958a-87be9d86be8f","Type":"ContainerDied","Data":"1c980f002c375b48bf2fa21cf7bb8a1e058307d6d03405427ea89318ae0d39b0"}
Nov 02 18:03:19.351290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.351274    4476 scope.go:122] "RemoveContainer" containerID="0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd"
Nov 02 18:03:19.364337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.364322    4476 scope.go:122] "RemoveContainer" containerID="2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"
Nov 02 18:03:19.374122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.374102    4476 scope.go:122] "RemoveContainer" containerID="0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd"
Nov 02 18:03:19.374394 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:19.374372    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd\": container with ID starting with 0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd not found: ID does not exist" containerID="0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd"
Nov 02 18:03:19.374469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.374407    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd"} err="failed to get container status \"0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd\": rpc error: code = NotFound desc = could not find container \"0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd\": container with ID starting with 0c4e3c01911bfd31a2cdd9f246feb44d8b32ad87a77e81e61357cbab87066ccd not found: ID does not exist"
Nov 02 18:03:19.374469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.374432    4476 scope.go:122] "RemoveContainer" containerID="2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"
Nov 02 18:03:19.374727 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:19.374701    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2\": container with ID starting with 2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2 not found: ID does not exist" containerID="2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"
Nov 02 18:03:19.374779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.374737    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2"} err="failed to get container status \"2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2\": rpc error: code = NotFound desc = could not find container \"2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2\": container with ID starting with 2e5daa44ff76b01f77f88b88762e6abdd0fad84ae4d8eb73ea8e42b3ce3b5ce2 not found: ID does not exist"
Nov 02 18:03:19.404735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.404689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:19.523032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.522995    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"]
Nov 02 18:03:19.570835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.570795    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ccf013-openshift-default-68966fc57f-jt7f9"]
Nov 02 18:03:19.630123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:19.630092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:21.389687 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:21.389621    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" path="/var/lib/kubelet/pods/ab0c4a68-ea6a-43ea-958a-87be9d86be8f/volumes"
Nov 02 18:03:22.173917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:22.173881    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"]
Nov 02 18:03:22.174188 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:22.174126    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" containerID="cri-o://36988d14b2aa650aaa964cd4f284b7e389b022c86049954e36fc105d6154d06f" gracePeriod=30
Nov 02 18:03:22.688220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:22.688187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:22.905265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:22.905233    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:23.244988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:23.244945    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:03:23.245173 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:23.245018    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:03:24.533782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:24.533752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:25.271851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:25.271818    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:25.800023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:25.799978    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb" podUID="9b7b2cce-a690-4133-8f81-add2738ca1f1" containerName="main" containerID="cri-o://ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf" gracePeriod=45
Nov 02 18:03:26.061372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.061309    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:03:26.122769 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.122739    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm\") pod \"9b7b2cce-a690-4133-8f81-add2738ca1f1\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") "
Nov 02 18:03:26.122962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.122808    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home\") pod \"9b7b2cce-a690-4133-8f81-add2738ca1f1\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") "
Nov 02 18:03:26.122962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.122915    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir\") pod \"9b7b2cce-a690-4133-8f81-add2738ca1f1\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") "
Nov 02 18:03:26.122962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.122945    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd\" (UniqueName: \"kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd\") pod \"9b7b2cce-a690-4133-8f81-add2738ca1f1\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") "
Nov 02 18:03:26.123129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123007    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs\") pod \"9b7b2cce-a690-4133-8f81-add2738ca1f1\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") "
Nov 02 18:03:26.123129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123038    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache\") pod \"9b7b2cce-a690-4133-8f81-add2738ca1f1\" (UID: \"9b7b2cce-a690-4133-8f81-add2738ca1f1\") "
Nov 02 18:03:26.123129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123073    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home" pod "9b7b2cce-a690-4133-8f81-add2738ca1f1" (UID: "9b7b2cce-a690-4133-8f81-add2738ca1f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:26.123263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir" pod "9b7b2cce-a690-4133-8f81-add2738ca1f1" (UID: "9b7b2cce-a690-4133-8f81-add2738ca1f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:26.123411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123360    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache" pod "9b7b2cce-a690-4133-8f81-add2738ca1f1" (UID: "9b7b2cce-a690-4133-8f81-add2738ca1f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:26.123566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123464    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:26.123566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123478    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:26.123566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.123488    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:26.125152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.125133    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm" pod "9b7b2cce-a690-4133-8f81-add2738ca1f1" (UID: "9b7b2cce-a690-4133-8f81-add2738ca1f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:26.125260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.125245    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs" pod "9b7b2cce-a690-4133-8f81-add2738ca1f1" (UID: "9b7b2cce-a690-4133-8f81-add2738ca1f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:26.125305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.125274    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd" pod "9b7b2cce-a690-4133-8f81-add2738ca1f1" (UID: "9b7b2cce-a690-4133-8f81-add2738ca1f1"). InnerVolumeSpecName "kube-api-access-gg4kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:26.225113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.225081    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4kd\" (UniqueName: \"kubernetes.io/projected/9b7b2cce-a690-4133-8f81-add2738ca1f1-kube-api-access-gg4kd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:26.225113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.225112    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7b2cce-a690-4133-8f81-add2738ca1f1-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:26.225302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.225122    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7b2cce-a690-4133-8f81-add2738ca1f1-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:26.403073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.403042    4476 generic.go:356] "Generic (PLEG): container finished" podID="9b7b2cce-a690-4133-8f81-add2738ca1f1" containerID="ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf" exitCode=0
Nov 02 18:03:26.403240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.403083    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"
Nov 02 18:03:26.403240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.403123    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb" event={"ID":"9b7b2cce-a690-4133-8f81-add2738ca1f1","Type":"ContainerDied","Data":"ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf"}
Nov 02 18:03:26.403240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.403167    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb" event={"ID":"9b7b2cce-a690-4133-8f81-add2738ca1f1","Type":"ContainerDied","Data":"7d375851c4353b8b8b3e85ad74e106da1c1ad5cef2bc15a876ab16d411b50fa2"}
Nov 02 18:03:26.403240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.403184    4476 scope.go:122] "RemoveContainer" containerID="ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf"
Nov 02 18:03:26.413685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.413669    4476 scope.go:122] "RemoveContainer" containerID="ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf"
Nov 02 18:03:26.413959 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:26.413940    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf\": container with ID starting with ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf not found: ID does not exist" containerID="ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf"
Nov 02 18:03:26.414002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.413969    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf"} err="failed to get container status \"ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf\": rpc error: code = NotFound desc = could not find container \"ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf\": container with ID starting with ab22103e99bdfaa7de4c2df2dde3620cf4dfb98977e0aa021a9c30bdeb5e8ebf not found: ID does not exist"
Nov 02 18:03:26.568751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.568714    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"]
Nov 02 18:03:26.616312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:26.616276    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-943ca1/praxis-route-943ca1-kserve-755b5b8f55-nqrgb"]
Nov 02 18:03:27.387697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.387656    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7b2cce-a690-4133-8f81-add2738ca1f1" path="/var/lib/kubelet/pods/9b7b2cce-a690-4133-8f81-add2738ca1f1/volumes"
Nov 02 18:03:27.413669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.413620    4476 generic.go:356] "Generic (PLEG): container finished" podID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerID="36988d14b2aa650aaa964cd4f284b7e389b022c86049954e36fc105d6154d06f" exitCode=0
Nov 02 18:03:27.413786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.413683    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" event={"ID":"e67d3c42-7784-450a-af12-5e0f6ee2c416","Type":"ContainerDied","Data":"36988d14b2aa650aaa964cd4f284b7e389b022c86049954e36fc105d6154d06f"}
Nov 02 18:03:27.434984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.434963    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 18:03:27.539112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539021    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539078    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539116    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539286    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539363    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539413    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539528    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:27.539645 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539564    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:27.539768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539748    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539802    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:27.539893 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539873    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.539958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.539908    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data\") pod \"e67d3c42-7784-450a-af12-5e0f6ee2c416\" (UID: \"e67d3c42-7784-450a-af12-5e0f6ee2c416\") "
Nov 02 18:03:27.540391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540276    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:27.540593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540568    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:27.540846 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540824    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67d3c42-7784-450a-af12-5e0f6ee2c416-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.540942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540854    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.540942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540869    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.540942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540883    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.540942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.540898    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.542388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.542350    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:27.542388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.542366    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:27.542553 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.542384    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "kube-api-access-5fsxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:27.542616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.542596    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy" pod "e67d3c42-7784-450a-af12-5e0f6ee2c416" (UID: "e67d3c42-7784-450a-af12-5e0f6ee2c416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:27.642430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.642399    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.642430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.642426    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.642430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.642438    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fsxs\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-kube-api-access-5fsxs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:27.642786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:27.642448    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67d3c42-7784-450a-af12-5e0f6ee2c416-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:28.423858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.423828    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"
Nov 02 18:03:28.424361 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.423871    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9" event={"ID":"e67d3c42-7784-450a-af12-5e0f6ee2c416","Type":"ContainerDied","Data":"042e364c6c4e04fb77832299768c16cb977fa86fd3ed06590bf854d0be387171"}
Nov 02 18:03:28.424361 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.423943    4476 scope.go:122] "RemoveContainer" containerID="36988d14b2aa650aaa964cd4f284b7e389b022c86049954e36fc105d6154d06f"
Nov 02 18:03:28.590221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.590189    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"]
Nov 02 18:03:28.614161 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.614111    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:28.640847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.640818    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-fxgf9"]
Nov 02 18:03:28.830095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.830020    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"]
Nov 02 18:03:28.830303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:28.830222    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" podUID="d5da7c40-e4c1-44cf-b471-8646c8901130" containerName="maas-api" containerID="cri-o://b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d" gracePeriod=30
Nov 02 18:03:29.018487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.018409    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:29.075112 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:29.074994    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:03:29.133569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.133543    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 18:03:29.260305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.260266    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s\" (UniqueName: \"kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s\") pod \"d5da7c40-e4c1-44cf-b471-8646c8901130\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") "
Nov 02 18:03:29.260984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.260955    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca\") pod \"d5da7c40-e4c1-44cf-b471-8646c8901130\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") "
Nov 02 18:03:29.261146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.261044    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\") pod \"d5da7c40-e4c1-44cf-b471-8646c8901130\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") "
Nov 02 18:03:29.261146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.261134    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs\") pod \"d5da7c40-e4c1-44cf-b471-8646c8901130\" (UID: \"d5da7c40-e4c1-44cf-b471-8646c8901130\") "
Nov 02 18:03:29.261867 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.261567    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca" pod "d5da7c40-e4c1-44cf-b471-8646c8901130" (UID: "d5da7c40-e4c1-44cf-b471-8646c8901130"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:29.274025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.273995    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs" pod "d5da7c40-e4c1-44cf-b471-8646c8901130" (UID: "d5da7c40-e4c1-44cf-b471-8646c8901130"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:29.274192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.274164    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s" pod "d5da7c40-e4c1-44cf-b471-8646c8901130" (UID: "d5da7c40-e4c1-44cf-b471-8646c8901130"). InnerVolumeSpecName "kube-api-access-x8r4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:29.275136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.275110    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls" pod "d5da7c40-e4c1-44cf-b471-8646c8901130" (UID: "d5da7c40-e4c1-44cf-b471-8646c8901130"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:29.341706 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.341661    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"]
Nov 02 18:03:29.341895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.341875    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb" podUID="c0469b71-13e4-48fc-a23d-0d0995410a9f" containerName="payload-processing" containerID="cri-o://9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144" gracePeriod=30
Nov 02 18:03:29.362088 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.362061    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.362208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.362090    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8r4s\" (UniqueName: \"kubernetes.io/projected/d5da7c40-e4c1-44cf-b471-8646c8901130-kube-api-access-x8r4s\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.362208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.362106    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5da7c40-e4c1-44cf-b471-8646c8901130-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.362208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.362119    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5da7c40-e4c1-44cf-b471-8646c8901130-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.383445 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.383414    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" path="/var/lib/kubelet/pods/e67d3c42-7784-450a-af12-5e0f6ee2c416/volumes"
Nov 02 18:03:29.401102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.401078    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"]
Nov 02 18:03:29.401262 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.401245    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9" podUID="79b1e1dd-980e-48f9-8b33-9446f18dd355" containerName="payload-pre-processing" containerID="cri-o://c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d" gracePeriod=30
Nov 02 18:03:29.431667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.431619    4476 generic.go:356] "Generic (PLEG): container finished" podID="d5da7c40-e4c1-44cf-b471-8646c8901130" containerID="b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d" exitCode=0
Nov 02 18:03:29.432040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.431685    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"
Nov 02 18:03:29.432040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.431709    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" event={"ID":"d5da7c40-e4c1-44cf-b471-8646c8901130","Type":"ContainerDied","Data":"b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d"}
Nov 02 18:03:29.432040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.431761    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn" event={"ID":"d5da7c40-e4c1-44cf-b471-8646c8901130","Type":"ContainerDied","Data":"d119730f446894d0571619c42d5790b17b13920e701cc1a50de8d717708e3e9b"}
Nov 02 18:03:29.432040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.431786    4476 scope.go:122] "RemoveContainer" containerID="b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d"
Nov 02 18:03:29.527991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.518691    4476 scope.go:122] "RemoveContainer" containerID="b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d"
Nov 02 18:03:29.527991 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:29.519182    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d\": container with ID starting with b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d not found: ID does not exist" containerID="b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d"
Nov 02 18:03:29.527991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.519251    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d"} err="failed to get container status \"b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d\": rpc error: code = NotFound desc = could not find container \"b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d\": container with ID starting with b6a5c8ad29f3803cb4c8cb9b228b132ab965270647cf70a63787a6a6a517084d not found: ID does not exist"
Nov 02 18:03:29.544640 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.544580    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:29.653407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.653374    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:03:29.698380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.698352    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:03:29.752561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.752524    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"]
Nov 02 18:03:29.763622 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.763598    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d5924c-7c5c9f5cf5-dtqxn"]
Nov 02 18:03:29.767595 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.767575    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume\") pod \"c0469b71-13e4-48fc-a23d-0d0995410a9f\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") "
Nov 02 18:03:29.767750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.767609    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd\" (UniqueName: \"kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd\") pod \"79b1e1dd-980e-48f9-8b33-9446f18dd355\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") "
Nov 02 18:03:29.767833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.767808    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54\" (UniqueName: \"kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54\") pod \"c0469b71-13e4-48fc-a23d-0d0995410a9f\" (UID: \"c0469b71-13e4-48fc-a23d-0d0995410a9f\") "
Nov 02 18:03:29.767903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.767889    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume\") pod \"79b1e1dd-980e-48f9-8b33-9446f18dd355\" (UID: \"79b1e1dd-980e-48f9-8b33-9446f18dd355\") "
Nov 02 18:03:29.768063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.768003    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume" pod "c0469b71-13e4-48fc-a23d-0d0995410a9f" (UID: "c0469b71-13e4-48fc-a23d-0d0995410a9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:29.768302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.768281    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0469b71-13e4-48fc-a23d-0d0995410a9f-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.768365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.768307    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume" pod "79b1e1dd-980e-48f9-8b33-9446f18dd355" (UID: "79b1e1dd-980e-48f9-8b33-9446f18dd355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:29.769909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.769890    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd" pod "79b1e1dd-980e-48f9-8b33-9446f18dd355" (UID: "79b1e1dd-980e-48f9-8b33-9446f18dd355"). InnerVolumeSpecName "kube-api-access-pttrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:29.770099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.770079    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54" pod "c0469b71-13e4-48fc-a23d-0d0995410a9f" (UID: "c0469b71-13e4-48fc-a23d-0d0995410a9f"). InnerVolumeSpecName "kube-api-access-24c54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:29.781436 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.781414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:29.869711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.869645    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b1e1dd-980e-48f9-8b33-9446f18dd355-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.869711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.869669    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttrd\" (UniqueName: \"kubernetes.io/projected/79b1e1dd-980e-48f9-8b33-9446f18dd355-kube-api-access-pttrd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:29.869711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:29.869681    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24c54\" (UniqueName: \"kubernetes.io/projected/c0469b71-13e4-48fc-a23d-0d0995410a9f-kube-api-access-24c54\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:30.441529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.441491    4476 generic.go:356] "Generic (PLEG): container finished" podID="c0469b71-13e4-48fc-a23d-0d0995410a9f" containerID="9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144" exitCode=0
Nov 02 18:03:30.442028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.441541    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"
Nov 02 18:03:30.442028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.441578    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb" event={"ID":"c0469b71-13e4-48fc-a23d-0d0995410a9f","Type":"ContainerDied","Data":"9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144"}
Nov 02 18:03:30.442028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.441654    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb" event={"ID":"c0469b71-13e4-48fc-a23d-0d0995410a9f","Type":"ContainerDied","Data":"1efa2efc5eea495dfef3989bd5f06babdbccfe632b0db25dde2e531ed39a3969"}
Nov 02 18:03:30.442028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.441661    4476 scope.go:122] "RemoveContainer" containerID="9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144"
Nov 02 18:03:30.443073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.443052    4476 generic.go:356] "Generic (PLEG): container finished" podID="79b1e1dd-980e-48f9-8b33-9446f18dd355" containerID="c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d" exitCode=0
Nov 02 18:03:30.443178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.443083    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"
Nov 02 18:03:30.443178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.443140    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9" event={"ID":"79b1e1dd-980e-48f9-8b33-9446f18dd355","Type":"ContainerDied","Data":"c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d"}
Nov 02 18:03:30.443264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.443190    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9" event={"ID":"79b1e1dd-980e-48f9-8b33-9446f18dd355","Type":"ContainerDied","Data":"0cbead8e8a79b6e1eb4669dbc37325d5b0adcbd3daaadfa09c312d9a72a54ec0"}
Nov 02 18:03:30.453471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.453454    4476 scope.go:122] "RemoveContainer" containerID="9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144"
Nov 02 18:03:30.453793 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:30.453766    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144\": container with ID starting with 9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144 not found: ID does not exist" containerID="9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144"
Nov 02 18:03:30.453856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.453805    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144"} err="failed to get container status \"9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144\": rpc error: code = NotFound desc = could not find container \"9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144\": container with ID starting with 9bf7284e1e854010a7b2b092452de56ff9e30a36aa036c82ce705982c174a144 not found: ID does not exist"
Nov 02 18:03:30.453856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.453830    4476 scope.go:122] "RemoveContainer" containerID="c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d"
Nov 02 18:03:30.462393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.462371    4476 scope.go:122] "RemoveContainer" containerID="c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d"
Nov 02 18:03:30.462660 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:30.462644    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d\": container with ID starting with c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d not found: ID does not exist" containerID="c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d"
Nov 02 18:03:30.462724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.462666    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d"} err="failed to get container status \"c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d\": rpc error: code = NotFound desc = could not find container \"c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d\": container with ID starting with c0067c2bc12107bd24fda28862ac2fc86fc5e9688f77e14a72efbfb7ae64ca3d not found: ID does not exist"
Nov 02 18:03:30.605508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.605463    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"]
Nov 02 18:03:30.608704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.608672    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-566576dc5f-ch7q9"]
Nov 02 18:03:30.696750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.696665    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:30.717288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.717261    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"]
Nov 02 18:03:30.722993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:30.722970    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-64dc45cd58-mzwkb"]
Nov 02 18:03:31.112546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:31.112520    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:31.382331 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:31.382258    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b1e1dd-980e-48f9-8b33-9446f18dd355" path="/var/lib/kubelet/pods/79b1e1dd-980e-48f9-8b33-9446f18dd355/volumes"
Nov 02 18:03:31.382670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:31.382657    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0469b71-13e4-48fc-a23d-0d0995410a9f" path="/var/lib/kubelet/pods/c0469b71-13e4-48fc-a23d-0d0995410a9f/volumes"
Nov 02 18:03:31.383016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:31.383004    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5da7c40-e4c1-44cf-b471-8646c8901130" path="/var/lib/kubelet/pods/d5da7c40-e4c1-44cf-b471-8646c8901130/volumes"
Nov 02 18:03:31.417886 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:31.417859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:32.003403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:32.003303    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"]
Nov 02 18:03:32.003941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:32.003713    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy" containerID="cri-o://6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1" gracePeriod=30
Nov 02 18:03:32.666366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:32.666332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:33.159764 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.159719    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:33.975744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.975710    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"]
Nov 02 18:03:33.976185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976171    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b65aa76e-6384-45f9-8b7a-97a919742be6" containerName="revoke-keys"
Nov 02 18:03:33.976233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976186    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65aa76e-6384-45f9-8b7a-97a919742be6" containerName="revoke-keys"
Nov 02 18:03:33.976233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976198    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy"
Nov 02 18:03:33.976233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976203    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy"
Nov 02 18:03:33.976233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976213    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2daa75d4-fa02-48e1-8981-e9704a3552ed" containerName="payload-pre-processing"
Nov 02 18:03:33.976233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976219    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2daa75d4-fa02-48e1-8981-e9704a3552ed" containerName="payload-pre-processing"
Nov 02 18:03:33.976233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976234    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" containerName="payload-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976239    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" containerName="payload-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976255    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976260    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976266    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9b7b2cce-a690-4133-8f81-add2738ca1f1" containerName="main"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976270    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7b2cce-a690-4133-8f81-add2738ca1f1" containerName="main"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976280    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976284    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976296    4476 cpu_manager.go:414] "containerMap: removing container" podUID="79b1e1dd-980e-48f9-8b33-9446f18dd355" containerName="payload-pre-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976301    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b1e1dd-980e-48f9-8b33-9446f18dd355" containerName="payload-pre-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976316    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d5da7c40-e4c1-44cf-b471-8646c8901130" containerName="maas-api"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976320    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5da7c40-e4c1-44cf-b471-8646c8901130" containerName="maas-api"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976326    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c0469b71-13e4-48fc-a23d-0d0995410a9f" containerName="payload-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976330    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0469b71-13e4-48fc-a23d-0d0995410a9f" containerName="payload-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976394    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976401    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b1e1dd-980e-48f9-8b33-9446f18dd355" containerName="payload-pre-processing"
Nov 02 18:03:33.976408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976410    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5da7c40-e4c1-44cf-b471-8646c8901130" containerName="maas-api"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976416    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0469b71-13e4-48fc-a23d-0d0995410a9f" containerName="payload-processing"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976426    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65aa76e-6384-45f9-8b7a-97a919742be6" containerName="revoke-keys"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976433    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0c4a68-ea6a-43ea-958a-87be9d86be8f" containerName="istio-proxy"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976438    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2daa75d4-fa02-48e1-8981-e9704a3552ed" containerName="payload-pre-processing"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976444    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6d359a-e2b8-49d6-baa3-a4fe7b23e8ed" containerName="payload-processing"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976451    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67d3c42-7784-450a-af12-5e0f6ee2c416" containerName="istio-proxy"
Nov 02 18:03:33.976881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.976455    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7b2cce-a690-4133-8f81-add2738ca1f1" containerName="main"
Nov 02 18:03:33.981318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:33.981300    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.036284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.036256    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"]
Nov 02 18:03:34.081804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.081779    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:03:34.082126 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.082104    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:03:34.114571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.114537    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.114731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.114590    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzt7\" (UniqueName: \"kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.114775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.114759    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.216227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.216190    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzt7\" (UniqueName: \"kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.216727 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.216262    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.216727 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.216335    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.216964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.216942    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.218833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.218812    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.273909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.273826    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzt7\" (UniqueName: \"kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7\") pod \"maas-api-revoke-sub-2041fbc1-9b9gc\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.294512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.294485    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:34.675518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:34.675495    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"]
Nov 02 18:03:34.676898 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:03:34.676868    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d6d1a8_cc1a_4130_8787_b6bbab8578e0.slice/crio-1a18680fca573f4c6f865f8bbd7e24509a6c90b6dfa7fb53c963cc4e51c7650c WatchSource:0}: Error finding container 1a18680fca573f4c6f865f8bbd7e24509a6c90b6dfa7fb53c963cc4e51c7650c: Status 404 returned error can't find the container with id 1a18680fca573f4c6f865f8bbd7e24509a6c90b6dfa7fb53c963cc4e51c7650c
Nov 02 18:03:35.481164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:35.481131    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc" event={"ID":"22d6d1a8-cc1a-4130-8787-b6bbab8578e0","Type":"ContainerStarted","Data":"3ac41bc5792aaf040f17c62a59dbe98a71e63ca1668f566f9ea13646e702633a"}
Nov 02 18:03:35.481565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:35.481187    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc" event={"ID":"22d6d1a8-cc1a-4130-8787-b6bbab8578e0","Type":"ContainerStarted","Data":"1a18680fca573f4c6f865f8bbd7e24509a6c90b6dfa7fb53c963cc4e51c7650c"}
Nov 02 18:03:35.597829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:35.597775    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc" podStartSLOduration=2.597760246 podStartE2EDuration="2.597760246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:35.597484709 +0000 UTC m=+2446.438090654" watchObservedRunningTime="2026-11-02 18:03:35.597760246 +0000 UTC m=+2446.438366181"
Nov 02 18:03:36.491411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:36.491371    4476 generic.go:356] "Generic (PLEG): container finished" podID="22d6d1a8-cc1a-4130-8787-b6bbab8578e0" containerID="3ac41bc5792aaf040f17c62a59dbe98a71e63ca1668f566f9ea13646e702633a" exitCode=0
Nov 02 18:03:36.491836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:36.491465    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc" event={"ID":"22d6d1a8-cc1a-4130-8787-b6bbab8578e0","Type":"ContainerDied","Data":"3ac41bc5792aaf040f17c62a59dbe98a71e63ca1668f566f9ea13646e702633a"}
Nov 02 18:03:36.544260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:36.544215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:36.941091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:36.941046    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:37.268092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.268067    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 18:03:37.350518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350487    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.350713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350543    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.350713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350616    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.350839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350818    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:37.350907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350848    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.350907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350899    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.351009 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.350939    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.351543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.351543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351142    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.351543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351155    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:37.351543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351172    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy\") pod \"3d5d2314-e566-4b04-92e8-08c64f93164b\" (UID: \"3d5d2314-e566-4b04-92e8-08c64f93164b\") "
Nov 02 18:03:37.351543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351305    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:37.351543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351442    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:37.352204 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351593    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.352204 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351615    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.352204 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351658    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.352204 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.351825    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:37.353198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.353174    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:37.353428 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.353406    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "kube-api-access-t26s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:37.353619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.353598    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:37.353768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.353751    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy" pod "3d5d2314-e566-4b04-92e8-08c64f93164b" (UID: "3d5d2314-e566-4b04-92e8-08c64f93164b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:37.452758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.452723    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.452758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.452756    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.452974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.452770    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.452974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.452787    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26s9\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-kube-api-access-t26s9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.452974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.452803    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5d2314-e566-4b04-92e8-08c64f93164b-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.452974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.452815    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5d2314-e566-4b04-92e8-08c64f93164b-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.499238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.499155    4476 generic.go:356] "Generic (PLEG): container finished" podID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerID="6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1" exitCode=0
Nov 02 18:03:37.499238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.499203    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"
Nov 02 18:03:37.499776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.499243    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" event={"ID":"3d5d2314-e566-4b04-92e8-08c64f93164b","Type":"ContainerDied","Data":"6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1"}
Nov 02 18:03:37.499776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.499297    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs" event={"ID":"3d5d2314-e566-4b04-92e8-08c64f93164b","Type":"ContainerDied","Data":"d46c4e5513df31213a29029e3f20dc4eac8d1b0f02e79d404d39d9472d0c1591"}
Nov 02 18:03:37.499776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.499323    4476 scope.go:122] "RemoveContainer" containerID="6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1"
Nov 02 18:03:37.509178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.509165    4476 scope.go:122] "RemoveContainer" containerID="6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1"
Nov 02 18:03:37.509464 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:37.509444    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1\": container with ID starting with 6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1 not found: ID does not exist" containerID="6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1"
Nov 02 18:03:37.509536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.509475    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1"} err="failed to get container status \"6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1\": rpc error: code = NotFound desc = could not find container \"6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1\": container with ID starting with 6a9b2e8d72a74723b0e100c7361d1091345de1ded08e5dee035e6ceb59659cd1 not found: ID does not exist"
Nov 02 18:03:37.650034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.650007    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:37.664352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.664322    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"]
Nov 02 18:03:37.672001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.671979    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-vm5xs"]
Nov 02 18:03:37.735594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.735561    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:37.755869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.755803    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca\") pod \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") "
Nov 02 18:03:37.756019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.755881    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7\" (UniqueName: \"kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7\") pod \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") "
Nov 02 18:03:37.756019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.755976    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca\") pod \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\" (UID: \"22d6d1a8-cc1a-4130-8787-b6bbab8578e0\") "
Nov 02 18:03:37.756235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.756208    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca" pod "22d6d1a8-cc1a-4130-8787-b6bbab8578e0" (UID: "22d6d1a8-cc1a-4130-8787-b6bbab8578e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:37.756333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.756299    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.758154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.758129    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7" pod "22d6d1a8-cc1a-4130-8787-b6bbab8578e0" (UID: "22d6d1a8-cc1a-4130-8787-b6bbab8578e0"). InnerVolumeSpecName "kube-api-access-fwzt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:37.758317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.758297    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca" pod "22d6d1a8-cc1a-4130-8787-b6bbab8578e0" (UID: "22d6d1a8-cc1a-4130-8787-b6bbab8578e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:37.856933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.856902    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzt7\" (UniqueName: \"kubernetes.io/projected/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-kube-api-access-fwzt7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.856933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.856934    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d6d1a8-cc1a-4130-8787-b6bbab8578e0-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:37.935823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:37.935787    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:38.508169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:38.508128    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"
Nov 02 18:03:38.508688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:38.508132    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc" event={"ID":"22d6d1a8-cc1a-4130-8787-b6bbab8578e0","Type":"ContainerDied","Data":"1a18680fca573f4c6f865f8bbd7e24509a6c90b6dfa7fb53c963cc4e51c7650c"}
Nov 02 18:03:38.508688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:38.508297    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a18680fca573f4c6f865f8bbd7e24509a6c90b6dfa7fb53c963cc4e51c7650c"
Nov 02 18:03:39.383724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:39.383682    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" path="/var/lib/kubelet/pods/3d5d2314-e566-4b04-92e8-08c64f93164b/volumes"
Nov 02 18:03:41.765718 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:41.765680    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:42.801015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:42.800979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:44.660511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.660475    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"]
Nov 02 18:03:44.661044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.661029    4476 cpu_manager.go:414] "containerMap: removing container" podUID="22d6d1a8-cc1a-4130-8787-b6bbab8578e0" containerName="revoke-keys"
Nov 02 18:03:44.661089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.661046    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d6d1a8-cc1a-4130-8787-b6bbab8578e0" containerName="revoke-keys"
Nov 02 18:03:44.661089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.661059    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy"
Nov 02 18:03:44.661089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.661065    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy"
Nov 02 18:03:44.661180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.661142    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d6d1a8-cc1a-4130-8787-b6bbab8578e0" containerName="revoke-keys"
Nov 02 18:03:44.661180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.661154    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5d2314-e566-4b04-92e8-08c64f93164b" containerName="istio-proxy"
Nov 02 18:03:44.664855 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.664831    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.721222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.721192    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"]
Nov 02 18:03:44.766772 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.766736    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:03:44.766931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.766777    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:03:44.769911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.769892    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"]
Nov 02 18:03:44.773714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.773695    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:44.830603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.830573    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.831585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.830647    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"]
Nov 02 18:03:44.831585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.830751    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.831585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.830811    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.932074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.931989    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.932074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.932046    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:44.932283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.932099    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:44.932283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.932245    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjm9f\" (UniqueName: \"kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:44.932394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.932292    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.932394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.932309    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.933160 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.933140    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.934528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.934506    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:44.990092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:44.990060    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9\") pod \"maas-api-revoke-sub-059e1cf5-b5w2t\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:45.033066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.033032    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.033066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.033065    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.033303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.033094    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjm9f\" (UniqueName: \"kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.033782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.033759    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.035584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.035566    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.089572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.089543    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjm9f\" (UniqueName: \"kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f\") pod \"maas-api-revoke-sub-c6ea676c-g989d\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.280521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.280422    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:45.389259 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.389233    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:45.658159 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.658130    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"]
Nov 02 18:03:45.659664 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:03:45.658879    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d34613_cfef_410c_9924_2af18cdc655b.slice/crio-378607c744e7d8ac14236286d52f5cd783f8565355869ab8971c7a591954c2e4 WatchSource:0}: Error finding container 378607c744e7d8ac14236286d52f5cd783f8565355869ab8971c7a591954c2e4: Status 404 returned error can't find the container with id 378607c744e7d8ac14236286d52f5cd783f8565355869ab8971c7a591954c2e4
Nov 02 18:03:45.768379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:45.768356    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"]
Nov 02 18:03:45.769328 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:03:45.769296    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90330102_9f35_4e13_a135_b22c26ad2b4f.slice/crio-1008f8a53435c2f86c287f021588f7ee3a4d6a4e8d1a49b8cf1de2ebd8558321 WatchSource:0}: Error finding container 1008f8a53435c2f86c287f021588f7ee3a4d6a4e8d1a49b8cf1de2ebd8558321: Status 404 returned error can't find the container with id 1008f8a53435c2f86c287f021588f7ee3a4d6a4e8d1a49b8cf1de2ebd8558321
Nov 02 18:03:46.399040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.398981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:46.587470 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.587425    4476 generic.go:356] "Generic (PLEG): container finished" podID="34d34613-cfef-410c-9924-2af18cdc655b" containerID="dd449cb6149af76e0666fcec0914e4537d9256b5f8ff2f7c0e17303c9c921546" exitCode=0
Nov 02 18:03:46.587679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.587544    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t" event={"ID":"34d34613-cfef-410c-9924-2af18cdc655b","Type":"ContainerDied","Data":"dd449cb6149af76e0666fcec0914e4537d9256b5f8ff2f7c0e17303c9c921546"}
Nov 02 18:03:46.587679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.587640    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t" event={"ID":"34d34613-cfef-410c-9924-2af18cdc655b","Type":"ContainerStarted","Data":"378607c744e7d8ac14236286d52f5cd783f8565355869ab8971c7a591954c2e4"}
Nov 02 18:03:46.589269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.589242    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d" event={"ID":"90330102-9f35-4e13-a135-b22c26ad2b4f","Type":"ContainerStarted","Data":"7e68cc6561ebafd1a9ca344379c479e7a5f5008675d90ff05c966915e0a93298"}
Nov 02 18:03:46.589399 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.589281    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d" event={"ID":"90330102-9f35-4e13-a135-b22c26ad2b4f","Type":"ContainerStarted","Data":"1008f8a53435c2f86c287f021588f7ee3a4d6a4e8d1a49b8cf1de2ebd8558321"}
Nov 02 18:03:46.814621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:46.814519    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d" podStartSLOduration=2.814504288 podStartE2EDuration="2.814504288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:46.813195787 +0000 UTC m=+2457.653801722" watchObservedRunningTime="2026-11-02 18:03:46.814504288 +0000 UTC m=+2457.655110215"
Nov 02 18:03:47.587249 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.587184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:47.602077 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.602041    4476 generic.go:356] "Generic (PLEG): container finished" podID="90330102-9f35-4e13-a135-b22c26ad2b4f" containerID="7e68cc6561ebafd1a9ca344379c479e7a5f5008675d90ff05c966915e0a93298" exitCode=0
Nov 02 18:03:47.602478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.602445    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d" event={"ID":"90330102-9f35-4e13-a135-b22c26ad2b4f","Type":"ContainerDied","Data":"7e68cc6561ebafd1a9ca344379c479e7a5f5008675d90ff05c966915e0a93298"}
Nov 02 18:03:47.758750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.758718    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:47.864235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.864204    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9\") pod \"34d34613-cfef-410c-9924-2af18cdc655b\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") "
Nov 02 18:03:47.864621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.864338    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca\") pod \"34d34613-cfef-410c-9924-2af18cdc655b\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") "
Nov 02 18:03:47.864621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.864375    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca\") pod \"34d34613-cfef-410c-9924-2af18cdc655b\" (UID: \"34d34613-cfef-410c-9924-2af18cdc655b\") "
Nov 02 18:03:47.864784 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.864760    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca" pod "34d34613-cfef-410c-9924-2af18cdc655b" (UID: "34d34613-cfef-410c-9924-2af18cdc655b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:47.867024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.867001    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca" pod "34d34613-cfef-410c-9924-2af18cdc655b" (UID: "34d34613-cfef-410c-9924-2af18cdc655b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:47.867129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.867048    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9" pod "34d34613-cfef-410c-9924-2af18cdc655b" (UID: "34d34613-cfef-410c-9924-2af18cdc655b"). InnerVolumeSpecName "kube-api-access-zn9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:47.965757 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.965720    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9j9\" (UniqueName: \"kubernetes.io/projected/34d34613-cfef-410c-9924-2af18cdc655b-kube-api-access-zn9j9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:47.965757 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.965755    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:47.965757 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:47.965767    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34d34613-cfef-410c-9924-2af18cdc655b-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:48.611866 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.611815    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t" event={"ID":"34d34613-cfef-410c-9924-2af18cdc655b","Type":"ContainerDied","Data":"378607c744e7d8ac14236286d52f5cd783f8565355869ab8971c7a591954c2e4"}
Nov 02 18:03:48.612082 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.611891    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378607c744e7d8ac14236286d52f5cd783f8565355869ab8971c7a591954c2e4"
Nov 02 18:03:48.612082 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.611825    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"
Nov 02 18:03:48.781072 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.781043    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:48.870706 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.870596    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"]
Nov 02 18:03:48.871300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.871281    4476 cpu_manager.go:414] "containerMap: removing container" podUID="34d34613-cfef-410c-9924-2af18cdc655b" containerName="revoke-keys"
Nov 02 18:03:48.871377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.871303    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d34613-cfef-410c-9924-2af18cdc655b" containerName="revoke-keys"
Nov 02 18:03:48.871377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.871334    4476 cpu_manager.go:414] "containerMap: removing container" podUID="90330102-9f35-4e13-a135-b22c26ad2b4f" containerName="revoke-keys"
Nov 02 18:03:48.871377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.871342    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90330102-9f35-4e13-a135-b22c26ad2b4f" containerName="revoke-keys"
Nov 02 18:03:48.871536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.871465    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="90330102-9f35-4e13-a135-b22c26ad2b4f" containerName="revoke-keys"
Nov 02 18:03:48.871536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.871486    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d34613-cfef-410c-9924-2af18cdc655b" containerName="revoke-keys"
Nov 02 18:03:48.874283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.874257    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca\") pod \"90330102-9f35-4e13-a135-b22c26ad2b4f\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") "
Nov 02 18:03:48.874414 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.874376    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca\") pod \"90330102-9f35-4e13-a135-b22c26ad2b4f\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") "
Nov 02 18:03:48.874496 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.874480    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f\" (UniqueName: \"kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f\") pod \"90330102-9f35-4e13-a135-b22c26ad2b4f\" (UID: \"90330102-9f35-4e13-a135-b22c26ad2b4f\") "
Nov 02 18:03:48.875008 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.874965    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca" pod "90330102-9f35-4e13-a135-b22c26ad2b4f" (UID: "90330102-9f35-4e13-a135-b22c26ad2b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:48.875375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.875356    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:48.876922 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.876897    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca" pod "90330102-9f35-4e13-a135-b22c26ad2b4f" (UID: "90330102-9f35-4e13-a135-b22c26ad2b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:48.880135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.880098    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f" pod "90330102-9f35-4e13-a135-b22c26ad2b4f" (UID: "90330102-9f35-4e13-a135-b22c26ad2b4f"). InnerVolumeSpecName "kube-api-access-hjm9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:48.934301 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.934259    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"]
Nov 02 18:03:48.973682 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.973650    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:48.975884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.975848    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:48.976032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.975899    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpms\" (UniqueName: \"kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:48.976088 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.976055    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:48.976239 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.976223    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:48.976277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.976246    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjm9f\" (UniqueName: \"kubernetes.io/projected/90330102-9f35-4e13-a135-b22c26ad2b4f-kube-api-access-hjm9f\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:48.976277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:48.976262    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90330102-9f35-4e13-a135-b22c26ad2b4f-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:49.077880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.077839    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.078064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.077910    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpms\" (UniqueName: \"kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.078064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.077962    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.078681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.078653    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.080525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.080501    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.134279 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.134214    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpms\" (UniqueName: \"kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms\") pod \"maas-api-revoke-sub-6e34c24e-b79zs\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.202212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.202185    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:49.579710 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.579675    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"]
Nov 02 18:03:49.580946 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:03:49.580920    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a0ac18_4849_4ac2_8e98_7d305a8a534f.slice/crio-16cd70f4ff2da232652c5d8b6d341afccd54c181a920c00e35e638415630e2a0 WatchSource:0}: Error finding container 16cd70f4ff2da232652c5d8b6d341afccd54c181a920c00e35e638415630e2a0: Status 404 returned error can't find the container with id 16cd70f4ff2da232652c5d8b6d341afccd54c181a920c00e35e638415630e2a0
Nov 02 18:03:49.620382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.620352    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs" event={"ID":"b2a0ac18-4849-4ac2-8e98-7d305a8a534f","Type":"ContainerStarted","Data":"16cd70f4ff2da232652c5d8b6d341afccd54c181a920c00e35e638415630e2a0"}
Nov 02 18:03:49.621619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.621601    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"
Nov 02 18:03:49.621750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.621601    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d" event={"ID":"90330102-9f35-4e13-a135-b22c26ad2b4f","Type":"ContainerDied","Data":"1008f8a53435c2f86c287f021588f7ee3a4d6a4e8d1a49b8cf1de2ebd8558321"}
Nov 02 18:03:49.621817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.621751    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1008f8a53435c2f86c287f021588f7ee3a4d6a4e8d1a49b8cf1de2ebd8558321"
Nov 02 18:03:49.771805 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.771703    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:49.980377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:49.980347    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:50.630408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:50.630372    4476 generic.go:356] "Generic (PLEG): container finished" podID="b2a0ac18-4849-4ac2-8e98-7d305a8a534f" containerID="57f9c7e74abb3c61e8a6043944da4c78fcbe190fa2f75405d96a4328114e67ae" exitCode=0
Nov 02 18:03:50.630557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:50.630433    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs" event={"ID":"b2a0ac18-4849-4ac2-8e98-7d305a8a534f","Type":"ContainerDied","Data":"57f9c7e74abb3c61e8a6043944da4c78fcbe190fa2f75405d96a4328114e67ae"}
Nov 02 18:03:51.550675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.550617    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:51.793060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.793038    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:51.806313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.806258    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms\" (UniqueName: \"kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms\") pod \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") "
Nov 02 18:03:51.806432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.806337    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca\") pod \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") "
Nov 02 18:03:51.806432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.806371    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca\") pod \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\" (UID: \"b2a0ac18-4849-4ac2-8e98-7d305a8a534f\") "
Nov 02 18:03:51.806781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.806757    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca" pod "b2a0ac18-4849-4ac2-8e98-7d305a8a534f" (UID: "b2a0ac18-4849-4ac2-8e98-7d305a8a534f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:51.808528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.808501    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms" pod "b2a0ac18-4849-4ac2-8e98-7d305a8a534f" (UID: "b2a0ac18-4849-4ac2-8e98-7d305a8a534f"). InnerVolumeSpecName "kube-api-access-kqpms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:51.808654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.808547    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca" pod "b2a0ac18-4849-4ac2-8e98-7d305a8a534f" (UID: "b2a0ac18-4849-4ac2-8e98-7d305a8a534f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:03:51.907272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.907193    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpms\" (UniqueName: \"kubernetes.io/projected/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-kube-api-access-kqpms\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:51.907272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.907225    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:51.907272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:51.907235    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a0ac18-4849-4ac2-8e98-7d305a8a534f-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:03:52.644093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:52.644054    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs" event={"ID":"b2a0ac18-4849-4ac2-8e98-7d305a8a534f","Type":"ContainerDied","Data":"16cd70f4ff2da232652c5d8b6d341afccd54c181a920c00e35e638415630e2a0"}
Nov 02 18:03:52.644093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:52.644097    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"
Nov 02 18:03:52.644678 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:52.644119    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16cd70f4ff2da232652c5d8b6d341afccd54c181a920c00e35e638415630e2a0"
Nov 02 18:03:53.734298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:53.734266    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:54.768290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:54.768256    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:58.205480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:03:58.205436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:59.060144 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:03:59.060102    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:04:00.505459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.505377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:00.830835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.830761    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"]
Nov 02 18:04:00.831219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.831205    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b2a0ac18-4849-4ac2-8e98-7d305a8a534f" containerName="revoke-keys"
Nov 02 18:04:00.831276 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.831221    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a0ac18-4849-4ac2-8e98-7d305a8a534f" containerName="revoke-keys"
Nov 02 18:04:00.831352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.831340    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a0ac18-4849-4ac2-8e98-7d305a8a534f" containerName="revoke-keys"
Nov 02 18:04:00.834907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.834886    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.893816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.893782    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg7f\" (UniqueName: \"kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.893991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.893914    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.893991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.893958    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.936733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.936703    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:04:00.936880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.936741    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:04:00.942079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.942052    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"]
Nov 02 18:04:00.994837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.994800    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.995031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.994856    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.995031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.994908    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg7f\" (UniqueName: \"kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.995498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.995477    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:00.997416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:00.997392    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:01.051526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:01.051498    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg7f\" (UniqueName: \"kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f\") pod \"maas-api-revoke-sub-33e36ef6-9mb2d\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:01.148845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:01.148807    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:01.532548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:01.532515    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"]
Nov 02 18:04:01.534067 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:04:01.534041    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2edd3e_3365_428d_8d24_a70dc5623451.slice/crio-88961a324d03e84b376af657759c0efc3f901373267293379b2cbdfd5c0805c8 WatchSource:0}: Error finding container 88961a324d03e84b376af657759c0efc3f901373267293379b2cbdfd5c0805c8: Status 404 returned error can't find the container with id 88961a324d03e84b376af657759c0efc3f901373267293379b2cbdfd5c0805c8
Nov 02 18:04:01.702437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:01.702403    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d" event={"ID":"3f2edd3e-3365-428d-8d24-a70dc5623451","Type":"ContainerStarted","Data":"88961a324d03e84b376af657759c0efc3f901373267293379b2cbdfd5c0805c8"}
Nov 02 18:04:02.710139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:02.710044    4476 generic.go:356] "Generic (PLEG): container finished" podID="3f2edd3e-3365-428d-8d24-a70dc5623451" containerID="6e5736406b52e31944e503df6efdc4c45d1219a146f5bc44ad90231d8d1bc2e2" exitCode=0
Nov 02 18:04:02.710513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:02.710137    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d" event={"ID":"3f2edd3e-3365-428d-8d24-a70dc5623451","Type":"ContainerDied","Data":"6e5736406b52e31944e503df6efdc4c45d1219a146f5bc44ad90231d8d1bc2e2"}
Nov 02 18:04:02.973476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:02.973385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:03.875327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.875296    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:03.922267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.922235    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f\" (UniqueName: \"kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f\") pod \"3f2edd3e-3365-428d-8d24-a70dc5623451\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") "
Nov 02 18:04:03.922444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.922341    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca\") pod \"3f2edd3e-3365-428d-8d24-a70dc5623451\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") "
Nov 02 18:04:03.922444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.922408    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca\") pod \"3f2edd3e-3365-428d-8d24-a70dc5623451\" (UID: \"3f2edd3e-3365-428d-8d24-a70dc5623451\") "
Nov 02 18:04:03.922748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.922714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca" pod "3f2edd3e-3365-428d-8d24-a70dc5623451" (UID: "3f2edd3e-3365-428d-8d24-a70dc5623451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:04:03.924486 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.924463    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f" pod "3f2edd3e-3365-428d-8d24-a70dc5623451" (UID: "3f2edd3e-3365-428d-8d24-a70dc5623451"). InnerVolumeSpecName "kube-api-access-rzg7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:04:03.924577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:03.924538    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca" pod "3f2edd3e-3365-428d-8d24-a70dc5623451" (UID: "3f2edd3e-3365-428d-8d24-a70dc5623451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:04:04.023591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:04.023506    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:04.023591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:04.023542    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2edd3e-3365-428d-8d24-a70dc5623451-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:04.023591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:04.023553    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzg7f\" (UniqueName: \"kubernetes.io/projected/3f2edd3e-3365-428d-8d24-a70dc5623451-kube-api-access-rzg7f\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:04.723949 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:04.723914    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"
Nov 02 18:04:04.724131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:04.723914    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d" event={"ID":"3f2edd3e-3365-428d-8d24-a70dc5623451","Type":"ContainerDied","Data":"88961a324d03e84b376af657759c0efc3f901373267293379b2cbdfd5c0805c8"}
Nov 02 18:04:04.724131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:04.724048    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88961a324d03e84b376af657759c0efc3f901373267293379b2cbdfd5c0805c8"
Nov 02 18:04:05.963098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:05.963060    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"]
Nov 02 18:04:05.963766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:05.963750    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3f2edd3e-3365-428d-8d24-a70dc5623451" containerName="revoke-keys"
Nov 02 18:04:05.963832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:05.963770    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2edd3e-3365-428d-8d24-a70dc5623451" containerName="revoke-keys"
Nov 02 18:04:05.963912 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:05.963901    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2edd3e-3365-428d-8d24-a70dc5623451" containerName="revoke-keys"
Nov 02 18:04:05.967919 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:05.967901    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.020070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.020037    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ceb1ae-openshift-default-dockercfg-pmptc\""
Nov 02 18:04:06.044540 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044499    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.044715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044592    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.044715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044649    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.044816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044783    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwq29\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.044852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044833    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.044953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044936    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.044989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.044967    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.045109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.045092    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.045149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.045134    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.073810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.073779    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"]
Nov 02 18:04:06.146491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146444    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.146491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146494    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.146795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146548    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.146795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146584    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.146795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146699    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.146795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146747    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.146795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146781    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.147073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146844    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwq29\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.147073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146881    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.147073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.146955    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.147073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.147010    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.147073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.147046    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.147360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.147216    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.148035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.148008    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.149818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.149791    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.150071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.150045    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.152981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.152957    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"]
Nov 02 18:04:06.157935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.157916    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.204234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.204205    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.204465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.204442    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwq29\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.215679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.210024    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:04:06.215679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.210545    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:04:06.216232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.216211    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"]
Nov 02 18:04:06.248480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.248441    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.248480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.248481    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btdz\" (UniqueName: \"kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.248793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.248678    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.283879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.283848    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:06.350796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.349531    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.350796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.349665    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.350796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.349703    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btdz\" (UniqueName: \"kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.351778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.351755    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.355654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.354069    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.407465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.407438    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btdz\" (UniqueName: \"kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz\") pod \"maas-api-revoke-sub-b0ee669b-skhbc\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.491092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.491013    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:06.716212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.716189    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"]
Nov 02 18:04:06.716794 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:04:06.716771    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e971b0_0a6b_4e39_b0a9_f2cf7192e6a0.slice/crio-a6b1e27003b185122d727e3bd19f1a98d0cf6d03d141f99917ce1620308a9d17 WatchSource:0}: Error finding container a6b1e27003b185122d727e3bd19f1a98d0cf6d03d141f99917ce1620308a9d17: Status 404 returned error can't find the container with id a6b1e27003b185122d727e3bd19f1a98d0cf6d03d141f99917ce1620308a9d17
Nov 02 18:04:06.719210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.719179    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:04:06.719295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.719240    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:04:06.719295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.719269    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:04:06.737596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.737572    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" event={"ID":"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0","Type":"ContainerStarted","Data":"a6b1e27003b185122d727e3bd19f1a98d0cf6d03d141f99917ce1620308a9d17"}
Nov 02 18:04:06.879877 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:06.879843    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"]
Nov 02 18:04:06.881661 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:04:06.881604    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6f0046_8f9b_4867_b403_0a997463f58d.slice/crio-eb599fd7439fbbe70f4b8548b016a6b4c8dff4401b2f8b57223399c6df40d3f2 WatchSource:0}: Error finding container eb599fd7439fbbe70f4b8548b016a6b4c8dff4401b2f8b57223399c6df40d3f2: Status 404 returned error can't find the container with id eb599fd7439fbbe70f4b8548b016a6b4c8dff4401b2f8b57223399c6df40d3f2
Nov 02 18:04:07.127980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:07.127948    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:07.777224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:07.777185    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc" event={"ID":"1d6f0046-8f9b-4867-b403-0a997463f58d","Type":"ContainerStarted","Data":"d7c1d9df575c3d15f59b415933a314d931efa0324d505c5c22ebf7301f613c76"}
Nov 02 18:04:07.777390 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:07.777256    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc" event={"ID":"1d6f0046-8f9b-4867-b403-0a997463f58d","Type":"ContainerStarted","Data":"eb599fd7439fbbe70f4b8548b016a6b4c8dff4401b2f8b57223399c6df40d3f2"}
Nov 02 18:04:07.779344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:07.779312    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" event={"ID":"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0","Type":"ContainerStarted","Data":"c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec"}
Nov 02 18:04:07.893373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:07.893307    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc" podStartSLOduration=1.893288229 podStartE2EDuration="1.893288229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:07.890703403 +0000 UTC m=+2478.731309337" watchObservedRunningTime="2026-11-02 18:04:07.893288229 +0000 UTC m=+2478.733894165"
Nov 02 18:04:08.012783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.012726    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podStartSLOduration=3.012712639 podStartE2EDuration="3.012712639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:08.010403458 +0000 UTC m=+2478.851009392" watchObservedRunningTime="2026-11-02 18:04:08.012712639 +0000 UTC m=+2478.853318572"
Nov 02 18:04:08.284818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.284778    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:08.286715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.286688    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 18:04:08.286837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.286742    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 18:04:08.788527 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.788483    4476 generic.go:356] "Generic (PLEG): container finished" podID="1d6f0046-8f9b-4867-b403-0a997463f58d" containerID="d7c1d9df575c3d15f59b415933a314d931efa0324d505c5c22ebf7301f613c76" exitCode=0
Nov 02 18:04:08.788751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.788569    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc" event={"ID":"1d6f0046-8f9b-4867-b403-0a997463f58d","Type":"ContainerDied","Data":"d7c1d9df575c3d15f59b415933a314d931efa0324d505c5c22ebf7301f613c76"}
Nov 02 18:04:08.912205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:08.912168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:09.284464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:09.284420    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 18:04:09.284726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:09.284486    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 18:04:10.008708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.008679    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:10.109998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.109970    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca\") pod \"1d6f0046-8f9b-4867-b403-0a997463f58d\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") "
Nov 02 18:04:10.110166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.110011    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca\") pod \"1d6f0046-8f9b-4867-b403-0a997463f58d\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") "
Nov 02 18:04:10.110166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.110056    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz\" (UniqueName: \"kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz\") pod \"1d6f0046-8f9b-4867-b403-0a997463f58d\" (UID: \"1d6f0046-8f9b-4867-b403-0a997463f58d\") "
Nov 02 18:04:10.110370 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.110348    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca" pod "1d6f0046-8f9b-4867-b403-0a997463f58d" (UID: "1d6f0046-8f9b-4867-b403-0a997463f58d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:04:10.110518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.110497    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:10.112332 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.112304    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca" pod "1d6f0046-8f9b-4867-b403-0a997463f58d" (UID: "1d6f0046-8f9b-4867-b403-0a997463f58d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:04:10.112420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.112367    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz" pod "1d6f0046-8f9b-4867-b403-0a997463f58d" (UID: "1d6f0046-8f9b-4867-b403-0a997463f58d"). InnerVolumeSpecName "kube-api-access-4btdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:04:10.211097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.211056    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f0046-8f9b-4867-b403-0a997463f58d-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:10.211097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.211095    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4btdz\" (UniqueName: \"kubernetes.io/projected/1d6f0046-8f9b-4867-b403-0a997463f58d-kube-api-access-4btdz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:10.284412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.284363    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 18:04:10.284674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.284427    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 18:04:10.802909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.802881    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"
Nov 02 18:04:10.803133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.802918    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc" event={"ID":"1d6f0046-8f9b-4867-b403-0a997463f58d","Type":"ContainerDied","Data":"eb599fd7439fbbe70f4b8548b016a6b4c8dff4401b2f8b57223399c6df40d3f2"}
Nov 02 18:04:10.803133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:10.802962    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb599fd7439fbbe70f4b8548b016a6b4c8dff4401b2f8b57223399c6df40d3f2"
Nov 02 18:04:11.284846 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:11.284811    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 18:04:11.285258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:11.284879    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 18:04:12.285157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:12.285120    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 18:04:12.285570 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:12.285174    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 18:04:13.284282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:13.284241    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 02 18:04:13.284497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:13.284310    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 02 18:04:13.568223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:13.568136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:13.773297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:13.773246    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:14.473319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:14.473288    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:14.473701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:14.473683    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:14.474821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:14.474799    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:04:15.945889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:15.945857    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:16.184786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:16.184755    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:17.601875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:17.601835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:18.174895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:18.174854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:18.711242 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:18.711146    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:19.025365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:19.025270    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:20.944782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:20.944748    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"]
Nov 02 18:04:20.945234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:20.945221    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1d6f0046-8f9b-4867-b403-0a997463f58d" containerName="revoke-keys"
Nov 02 18:04:20.945278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:20.945236    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6f0046-8f9b-4867-b403-0a997463f58d" containerName="revoke-keys"
Nov 02 18:04:20.945316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:20.945304    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6f0046-8f9b-4867-b403-0a997463f58d" containerName="revoke-keys"
Nov 02 18:04:20.948686 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:20.948670    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.005586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.005556    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"]
Nov 02 18:04:21.050770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.050730    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:04:21.050968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.050781    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:04:21.061824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.061798    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.061962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.061827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg98c\" (UniqueName: \"kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.062047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.062022    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.163403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.163374    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.163612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.163469    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.163612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.163488    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg98c\" (UniqueName: \"kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.164181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.164159    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.166202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.166170    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.220664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.220556    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg98c\" (UniqueName: \"kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c\") pod \"maas-api-revoke-sub-9a35fbb9-cvqlf\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.265751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.265717    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:21.645044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.645015    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"]
Nov 02 18:04:21.646229 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:04:21.646203    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ab28ef_c8a1_490a_bcea_6be0b0dc9b1d.slice/crio-0612ea743fce28c18c85da97fa7a87702069dd0b35e00f3a611c6e75da83f7d7 WatchSource:0}: Error finding container 0612ea743fce28c18c85da97fa7a87702069dd0b35e00f3a611c6e75da83f7d7: Status 404 returned error can't find the container with id 0612ea743fce28c18c85da97fa7a87702069dd0b35e00f3a611c6e75da83f7d7
Nov 02 18:04:21.877763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:21.877731    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf" event={"ID":"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d","Type":"ContainerStarted","Data":"0612ea743fce28c18c85da97fa7a87702069dd0b35e00f3a611c6e75da83f7d7"}
Nov 02 18:04:22.885049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:22.885012    4476 generic.go:356] "Generic (PLEG): container finished" podID="09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" containerID="6b0c55788d4f0346fcf74f8891eb556d2bff1abb2c23326f36a256046cd1d876" exitCode=0
Nov 02 18:04:22.885442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:22.885097    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf" event={"ID":"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d","Type":"ContainerDied","Data":"6b0c55788d4f0346fcf74f8891eb556d2bff1abb2c23326f36a256046cd1d876"}
Nov 02 18:04:23.784360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:23.784312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:24.088967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.088941    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:24.098104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.098080    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca\") pod \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") "
Nov 02 18:04:24.098256 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.098210    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c\" (UniqueName: \"kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c\") pod \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") "
Nov 02 18:04:24.098256 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.098249    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca\") pod \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\" (UID: \"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d\") "
Nov 02 18:04:24.098848 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.098733    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca" pod "09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" (UID: "09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:04:24.100388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.100357    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca" pod "09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" (UID: "09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:04:24.100506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.100434    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c" pod "09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" (UID: "09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d"). InnerVolumeSpecName "kube-api-access-sg98c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:04:24.199667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.199610    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg98c\" (UniqueName: \"kubernetes.io/projected/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-kube-api-access-sg98c\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:24.199667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.199662    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:24.199868 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.199677    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:04:24.903151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.903124    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"
Nov 02 18:04:24.903350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.903167    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf" event={"ID":"09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d","Type":"ContainerDied","Data":"0612ea743fce28c18c85da97fa7a87702069dd0b35e00f3a611c6e75da83f7d7"}
Nov 02 18:04:24.903350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.903226    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0612ea743fce28c18c85da97fa7a87702069dd0b35e00f3a611c6e75da83f7d7"
Nov 02 18:04:24.938662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:24.938613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:26.447726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:26.447687    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:26.645315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:26.645225    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:27.214398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:27.214366    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:27.416618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:27.416587    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:28.468295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:28.468254    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:29.066536 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:04:29.066503    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:04:30.390056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:30.390024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:31.169149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:31.169112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:33.145901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:33.145866    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:33.342024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:33.341992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:34.687438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:34.687399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:35.172124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:35.172086    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:35.567415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:35.567330    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:39.361502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:39.361473    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:40.587312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:40.587278    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:40.800501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:40.800456    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:45.830004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:45.829967    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:46.195014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:46.194939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:46.388515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:46.388483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:46.793137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:46.793098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:51.326480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.326443    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"]
Nov 02 18:04:51.328096 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.328067    4476 cpu_manager.go:414] "containerMap: removing container" podUID="09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" containerName="revoke-keys"
Nov 02 18:04:51.328278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.328261    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" containerName="revoke-keys"
Nov 02 18:04:51.328621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.328604    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" containerName="revoke-keys"
Nov 02 18:04:51.333194 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.333174    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.441612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.441576    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"]
Nov 02 18:04:51.490439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490395    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.490439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490438    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.490702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490611    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.490767 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490746    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.490846 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490825    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.490922 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490901    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.491014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.490998    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khccr\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.491081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.491035    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.491081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.491068    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.592795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592702    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.592795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592751    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.592795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592790    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592834    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592869    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592895    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592958    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khccr\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.592990    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.593021    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.593315    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.593476    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.593584    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.593751    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.593985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.593955    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.595408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.595377    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.595524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.595408    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.651933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.651897    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.652261 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.652238    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khccr\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:51.948948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:51.948905    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:52.395104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:52.395060    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"]
Nov 02 18:04:52.395485 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:04:52.395203    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e3f0d_9dae_4657_ab2e_de705905edf7.slice/crio-95fa01db789a93f175f3c4e715fb38f61cd37bfdfd545abba26ae065c2656931 WatchSource:0}: Error finding container 95fa01db789a93f175f3c4e715fb38f61cd37bfdfd545abba26ae065c2656931: Status 404 returned error can't find the container with id 95fa01db789a93f175f3c4e715fb38f61cd37bfdfd545abba26ae065c2656931
Nov 02 18:04:52.398737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:52.398697    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:04:52.398832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:52.398785    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:04:52.398871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:52.398830    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:04:52.421986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:52.421951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:52.751422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:52.751313    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:53.103663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:53.103546    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" event={"ID":"e04e3f0d-9dae-4657-ab2e-de705905edf7","Type":"ContainerStarted","Data":"e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6"}
Nov 02 18:04:53.103663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:53.103611    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" event={"ID":"e04e3f0d-9dae-4657-ab2e-de705905edf7","Type":"ContainerStarted","Data":"95fa01db789a93f175f3c4e715fb38f61cd37bfdfd545abba26ae065c2656931"}
Nov 02 18:04:53.323541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:53.323483    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podStartSLOduration=2.323466165 podStartE2EDuration="2.323466165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:53.272852497 +0000 UTC m=+2524.113458432" watchObservedRunningTime="2026-11-02 18:04:53.323466165 +0000 UTC m=+2524.164072101"
Nov 02 18:04:53.959697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:53.958667    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:53.959697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:53.958741    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 18:04:53.959697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:53.958797    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 18:04:54.065022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:54.064981    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:54.951652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:54.951028    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 18:04:54.951652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:54.951091    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 18:04:55.950022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:55.949984    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 18:04:55.950728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:55.950692    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 18:04:56.954656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:56.951769    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 18:04:56.954656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:56.951831    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 18:04:57.955658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:57.952996    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 18:04:57.955658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:57.953056    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 18:04:58.178111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:58.176229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:58.950100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:58.950063    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 02 18:04:58.950278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:58.950130    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 02 18:04:59.061137 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:04:59.061096    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:04:59.954175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:59.954147    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:59.954603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:59.954580    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:04:59.955316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:04:59.955295    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:05:03.182647 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:03.182581    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:08.116223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:08.116189    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:10.344393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:10.344360    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:12.264984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:12.264937    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:14.076436 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:14.076399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:16.998568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:16.998536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:17.361860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:17.361819    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:17.611527 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:17.611496    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:18.726657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:18.726596    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:19.485836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:19.485765    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:21.405521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.405487    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"]
Nov 02 18:05:21.409299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.409283    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.469545 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.469510    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"]
Nov 02 18:05:21.513072 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.513040    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:05:21.513233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.513135    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:05:21.521129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.521104    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dst\" (UniqueName: \"kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.521280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.521176    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.521356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.521285    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.622055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.622024    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.622225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.622065    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.622225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.622139    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dst\" (UniqueName: \"kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.622660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.622620    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.624579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.624556    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.675197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.675113    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:21.685451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.685422    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dst\" (UniqueName: \"kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst\") pod \"maas-api-revoke-sub-8b6c6a0c-fdtrp\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.722865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.722838    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:21.881295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:21.881264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:22.103524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:22.103494    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"]
Nov 02 18:05:22.104291 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:05:22.104264    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99151887_76db_4d9d_a040_5a4e9c275caa.slice/crio-b41c2e66a3aefcbc5affb7d1dd1418922a1e29355a926c81090668cd6f2338e2 WatchSource:0}: Error finding container b41c2e66a3aefcbc5affb7d1dd1418922a1e29355a926c81090668cd6f2338e2: Status 404 returned error can't find the container with id b41c2e66a3aefcbc5affb7d1dd1418922a1e29355a926c81090668cd6f2338e2
Nov 02 18:05:22.308316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:22.308277    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp" event={"ID":"99151887-76db-4d9d-a040-5a4e9c275caa","Type":"ContainerStarted","Data":"b41c2e66a3aefcbc5affb7d1dd1418922a1e29355a926c81090668cd6f2338e2"}
Nov 02 18:05:22.964019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:22.963989    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"]
Nov 02 18:05:22.968277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:22.968253    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.027714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.027673    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"]
Nov 02 18:05:23.044948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.044915    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.044994    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045061    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045115    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045195    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045233    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045260    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82lf\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045294    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.045533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.045367    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.072424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.072390    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\""
Nov 02 18:05:23.146062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146027    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146072    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146119    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146165    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146201    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146231    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82lf\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146269    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146337    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146394    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146821    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.146943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146921    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.147007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146940    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.147007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.146959    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.147116 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.147074    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.149002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.148977    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.149482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.149459    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.203096 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.203018    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:23.203867 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.203841    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82lf\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.204335 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.204312    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.284092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.284055    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:23.319796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.319762    4476 generic.go:356] "Generic (PLEG): container finished" podID="99151887-76db-4d9d-a040-5a4e9c275caa" containerID="85e3f6eb15d5f27bf694a8259e9d3dfe5fdb45845dd9b7ec5674ebe29faff638" exitCode=0
Nov 02 18:05:23.319951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.319810    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp" event={"ID":"99151887-76db-4d9d-a040-5a4e9c275caa","Type":"ContainerDied","Data":"85e3f6eb15d5f27bf694a8259e9d3dfe5fdb45845dd9b7ec5674ebe29faff638"}
Nov 02 18:05:23.750425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.750391    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"]
Nov 02 18:05:23.752925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.752893    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:05:23.753036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.752957    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:05:23.753036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.752985    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:05:23.883951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:23.883911    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:24.237005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.236972    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:24.328185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.328147    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" event={"ID":"af345ff6-884c-4b2e-8560-3866eb898f54","Type":"ContainerStarted","Data":"448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476"}
Nov 02 18:05:24.328393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.328208    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" event={"ID":"af345ff6-884c-4b2e-8560-3866eb898f54","Type":"ContainerStarted","Data":"f78e57395ea45da414427efb408196facfe3375d1cc581c5cfee6d5783f558d4"}
Nov 02 18:05:24.451277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.451223    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podStartSLOduration=2.451204142 podStartE2EDuration="2.451204142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:24.446685098 +0000 UTC m=+2555.287291033" watchObservedRunningTime="2026-11-02 18:05:24.451204142 +0000 UTC m=+2555.291810077"
Nov 02 18:05:24.476821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.476779    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:24.570949 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.570849    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst\" (UniqueName: \"kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst\") pod \"99151887-76db-4d9d-a040-5a4e9c275caa\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") "
Nov 02 18:05:24.571148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.570973    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca\") pod \"99151887-76db-4d9d-a040-5a4e9c275caa\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") "
Nov 02 18:05:24.571148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.571033    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca\") pod \"99151887-76db-4d9d-a040-5a4e9c275caa\" (UID: \"99151887-76db-4d9d-a040-5a4e9c275caa\") "
Nov 02 18:05:24.571775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.571738    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca" pod "99151887-76db-4d9d-a040-5a4e9c275caa" (UID: "99151887-76db-4d9d-a040-5a4e9c275caa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:24.573768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.573730    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst" pod "99151887-76db-4d9d-a040-5a4e9c275caa" (UID: "99151887-76db-4d9d-a040-5a4e9c275caa"). InnerVolumeSpecName "kube-api-access-t6dst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:24.573887 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.573790    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca" pod "99151887-76db-4d9d-a040-5a4e9c275caa" (UID: "99151887-76db-4d9d-a040-5a4e9c275caa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:05:24.674654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.672372    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:24.674654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.672415    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99151887-76db-4d9d-a040-5a4e9c275caa-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:24.674654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:24.672434    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6dst\" (UniqueName: \"kubernetes.io/projected/99151887-76db-4d9d-a040-5a4e9c275caa-kube-api-access-t6dst\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:25.285146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:25.285100    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:25.287152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:25.287127    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:05:25.287281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:25.287185    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 02 18:05:25.336545 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:25.336516    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"
Nov 02 18:05:25.336719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:25.336552    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp" event={"ID":"99151887-76db-4d9d-a040-5a4e9c275caa","Type":"ContainerDied","Data":"b41c2e66a3aefcbc5affb7d1dd1418922a1e29355a926c81090668cd6f2338e2"}
Nov 02 18:05:25.336719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:25.336602    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b41c2e66a3aefcbc5affb7d1dd1418922a1e29355a926c81090668cd6f2338e2"
Nov 02 18:05:26.285164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:26.285128    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:05:26.285554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:26.285179    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 02 18:05:27.285161 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:27.285126    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:05:27.285319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:27.285194    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 02 18:05:28.284700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:28.284660    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:05:28.284881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:28.284714    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 02 18:05:28.562706 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:28.562287    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:29.077611 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:05:29.077573    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:05:29.285177 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:29.285134    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:05:29.285343 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:29.285193    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 02 18:05:30.284719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:30.284676    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 02 18:05:30.285211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:30.284744    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 02 18:05:30.506244 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:30.506156    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:31.304201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:31.304167    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:31.304670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:31.304655    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:31.307554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:31.307527    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:05:34.413064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:34.413019    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:34.650226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:34.650185    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:36.055556 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.055508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:36.193047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.193016    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"]
Nov 02 18:05:36.193524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.193510    4476 cpu_manager.go:414] "containerMap: removing container" podUID="99151887-76db-4d9d-a040-5a4e9c275caa" containerName="revoke-keys"
Nov 02 18:05:36.193590 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.193525    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99151887-76db-4d9d-a040-5a4e9c275caa" containerName="revoke-keys"
Nov 02 18:05:36.193654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.193607    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="99151887-76db-4d9d-a040-5a4e9c275caa" containerName="revoke-keys"
Nov 02 18:05:36.197177 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.197157    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.255835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.255805    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"]
Nov 02 18:05:36.295182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.295149    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.295380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.295194    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrtg\" (UniqueName: \"kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.295380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.295363    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.300205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.300182    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:05:36.300285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.300196    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:05:36.397724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.396782    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.397724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.396840    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrtg\" (UniqueName: \"kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.397724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.396952    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.399667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.398338    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.400839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.400782    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.442377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.442327    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"]
Nov 02 18:05:36.442604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.442572    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" containerID="cri-o://d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79" gracePeriod=30
Nov 02 18:05:36.455211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.455170    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrtg\" (UniqueName: \"kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg\") pod \"maas-api-revoke-sub-5079665e-k9mlg\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.512017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.511989    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:36.534537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.534496    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:36.697386 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.697309    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"]
Nov 02 18:05:36.702292 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.702276    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.784989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.784957    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"]
Nov 02 18:05:36.801387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801354    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801400    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnh9m\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801431    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801535    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801594    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801802 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801744    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801854 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801818    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801878    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.801913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.801905    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.902586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902549    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.902845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902608    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnh9m\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.902845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902693    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.902845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902718    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.902845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902761    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.902845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902805    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902856    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902911    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.902945    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.903078    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903345 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.903222    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903345 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.903265    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.903588    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.903723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.903694    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.905296 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.905274    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.905604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.905585    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.912136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.912113    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"]
Nov 02 18:05:36.913068 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:05:36.913046    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8340cef_b8e3_435e_9bfd_e2b87e67420d.slice/crio-540c84720e15803f5af27e22ba9a0ceb210d2753fa1b6cd11028503459cca55a WatchSource:0}: Error finding container 540c84720e15803f5af27e22ba9a0ceb210d2753fa1b6cd11028503459cca55a: Status 404 returned error can't find the container with id 540c84720e15803f5af27e22ba9a0ceb210d2753fa1b6cd11028503459cca55a
Nov 02 18:05:36.960351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.960324    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:36.960539 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:36.960403    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnh9m\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:37.017397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.017359    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:37.458122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.458099    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"]
Nov 02 18:05:37.459701 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:05:37.459672    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1449759_d1a4_46c9_a8bc_f5a71fda118b.slice/crio-4f478445ac75854a40f7daf7b39ad55dfddf14e5dd2a357037f1cf111a9b28bc WatchSource:0}: Error finding container 4f478445ac75854a40f7daf7b39ad55dfddf14e5dd2a357037f1cf111a9b28bc: Status 404 returned error can't find the container with id 4f478445ac75854a40f7daf7b39ad55dfddf14e5dd2a357037f1cf111a9b28bc
Nov 02 18:05:37.462275 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.462241    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:05:37.462389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.462309    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:05:37.462389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.462340    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:05:37.463035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.463017    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:37.486368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.486317    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg" event={"ID":"c8340cef-b8e3-435e-9bfd-e2b87e67420d","Type":"ContainerStarted","Data":"5b097ed691d5ea8f33d97f48c883ad30906b21069e97820dc2cd2b6b1c67a62b"}
Nov 02 18:05:37.486522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.486398    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg" event={"ID":"c8340cef-b8e3-435e-9bfd-e2b87e67420d","Type":"ContainerStarted","Data":"540c84720e15803f5af27e22ba9a0ceb210d2753fa1b6cd11028503459cca55a"}
Nov 02 18:05:37.488427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.488392    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" event={"ID":"c1449759-d1a4-46c9-a8bc-f5a71fda118b","Type":"ContainerStarted","Data":"4f478445ac75854a40f7daf7b39ad55dfddf14e5dd2a357037f1cf111a9b28bc"}
Nov 02 18:05:37.601775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:37.601717    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg" podStartSLOduration=1.601699522 podStartE2EDuration="1.601699522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:37.601413364 +0000 UTC m=+2568.442019301" watchObservedRunningTime="2026-11-02 18:05:37.601699522 +0000 UTC m=+2568.442305452"
Nov 02 18:05:38.497195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:38.497160    4476 generic.go:356] "Generic (PLEG): container finished" podID="c8340cef-b8e3-435e-9bfd-e2b87e67420d" containerID="5b097ed691d5ea8f33d97f48c883ad30906b21069e97820dc2cd2b6b1c67a62b" exitCode=0
Nov 02 18:05:38.497685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:38.497240    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg" event={"ID":"c8340cef-b8e3-435e-9bfd-e2b87e67420d","Type":"ContainerDied","Data":"5b097ed691d5ea8f33d97f48c883ad30906b21069e97820dc2cd2b6b1c67a62b"}
Nov 02 18:05:38.499024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:38.498999    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" event={"ID":"c1449759-d1a4-46c9-a8bc-f5a71fda118b","Type":"ContainerStarted","Data":"96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01"}
Nov 02 18:05:38.764736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:38.755654    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podStartSLOduration=2.755613448 podStartE2EDuration="2.755613448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:38.754123016 +0000 UTC m=+2569.594728955" watchObservedRunningTime="2026-11-02 18:05:38.755613448 +0000 UTC m=+2569.596219383"
Nov 02 18:05:39.017606 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.017516    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:39.019278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.019256    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:05:39.019342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.019304    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:05:39.702322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.702298    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:39.834995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.834916    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg\" (UniqueName: \"kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg\") pod \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") "
Nov 02 18:05:39.834995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.834987    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca\") pod \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") "
Nov 02 18:05:39.835226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.835022    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca\") pod \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\" (UID: \"c8340cef-b8e3-435e-9bfd-e2b87e67420d\") "
Nov 02 18:05:39.835413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.835384    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca" pod "c8340cef-b8e3-435e-9bfd-e2b87e67420d" (UID: "c8340cef-b8e3-435e-9bfd-e2b87e67420d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:39.837234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.837210    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca" pod "c8340cef-b8e3-435e-9bfd-e2b87e67420d" (UID: "c8340cef-b8e3-435e-9bfd-e2b87e67420d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:05:39.837411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.837343    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg" pod "c8340cef-b8e3-435e-9bfd-e2b87e67420d" (UID: "c8340cef-b8e3-435e-9bfd-e2b87e67420d"). InnerVolumeSpecName "kube-api-access-pwrtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:39.936745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.936706    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:39.936745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.936739    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8340cef-b8e3-435e-9bfd-e2b87e67420d-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:39.936745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:39.936752    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwrtg\" (UniqueName: \"kubernetes.io/projected/c8340cef-b8e3-435e-9bfd-e2b87e67420d-kube-api-access-pwrtg\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:40.018211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.018173    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:05:40.018371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.018235    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:05:40.386313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.385222    4476 patch_prober.go:29] interesting pod/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:05:40.386313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.385286    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:05:40.515066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.515036    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"
Nov 02 18:05:40.515268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.515035    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg" event={"ID":"c8340cef-b8e3-435e-9bfd-e2b87e67420d","Type":"ContainerDied","Data":"540c84720e15803f5af27e22ba9a0ceb210d2753fa1b6cd11028503459cca55a"}
Nov 02 18:05:40.515268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:40.515157    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="540c84720e15803f5af27e22ba9a0ceb210d2753fa1b6cd11028503459cca55a"
Nov 02 18:05:41.018358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.018322    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:05:41.018780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.018374    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:05:41.808340 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.808313    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991587    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991736    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991789    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991821    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991860    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991893    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991921    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991964    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.991992    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert\") pod \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\" (UID: \"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb\") "
Nov 02 18:05:41.992679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.992564    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:41.993313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.992883    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:41.993313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.993119    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:41.993419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.993371    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:41.993419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.993389    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:41.993419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.993405    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:41.993723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.993697    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:41.994117 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.993828    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:41.996873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.996424    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:05:41.998992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.998957    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:41.999678 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.999644    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "kube-api-access-256vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:41.999860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:41.999844    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token" pod "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" (UID: "0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:42.017902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.017874    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:05:42.018025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.017940    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:05:42.094415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.094336    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:42.094415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.094364    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:42.094415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.094375    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-256vh\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-kube-api-access-256vh\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:42.094415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.094385    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:42.094415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.094393    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:42.094415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.094403    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:05:42.524368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.524329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:42.531569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.531537    4476 generic.go:356] "Generic (PLEG): container finished" podID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerID="d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79" exitCode=0
Nov 02 18:05:42.531775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.531584    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"
Nov 02 18:05:42.531775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.531619    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" event={"ID":"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb","Type":"ContainerDied","Data":"d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79"}
Nov 02 18:05:42.531775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.531695    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8" event={"ID":"0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb","Type":"ContainerDied","Data":"8b49fcc46583d6fbc2abfaf3e5056dca0ba248c00516052d2abb6912cbb39d71"}
Nov 02 18:05:42.531775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.531718    4476 scope.go:122] "RemoveContainer" containerID="d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79"
Nov 02 18:05:42.574000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.573978    4476 scope.go:122] "RemoveContainer" containerID="5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"
Nov 02 18:05:42.587559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.587533    4476 scope.go:122] "RemoveContainer" containerID="d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79"
Nov 02 18:05:42.587998 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:05:42.587972    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79\": container with ID starting with d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79 not found: ID does not exist" containerID="d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79"
Nov 02 18:05:42.588059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.588014    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79"} err="failed to get container status \"d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79\": rpc error: code = NotFound desc = could not find container \"d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79\": container with ID starting with d6a575f58622f8ebc9e90d9b3d79bf9cb37db942aed4a65817c998df3f446b79 not found: ID does not exist"
Nov 02 18:05:42.588059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.588038    4476 scope.go:122] "RemoveContainer" containerID="5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"
Nov 02 18:05:42.588393 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:05:42.588354    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863\": container with ID starting with 5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863 not found: ID does not exist" containerID="5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"
Nov 02 18:05:42.588393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.588376    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863"} err="failed to get container status \"5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863\": rpc error: code = NotFound desc = could not find container \"5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863\": container with ID starting with 5ee0bb5a28e3254fa9c3171a4c3718acaa7d1543b360e0500fd2807d7723d863 not found: ID does not exist"
Nov 02 18:05:42.697777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.697745    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"]
Nov 02 18:05:42.703729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:42.703698    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2289f-openshift-default-585656b69c-8b7z8"]
Nov 02 18:05:43.017742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:43.017704    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:05:43.017897 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:43.017757    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:05:43.384080 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:43.384038    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" path="/var/lib/kubelet/pods/0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb/volumes"
Nov 02 18:05:44.018405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:44.018373    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:05:44.018572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:44.018426    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:05:45.022271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:45.022239    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:45.022781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:45.022674    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:45.023369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:45.023349    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:05:47.336620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:47.336579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:48.709451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:48.709397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:49.474258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:49.474225    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:49.715798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:49.715722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:52.462579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:52.462544    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:54.145934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:54.145902    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:54.394139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:54.394107    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:54.743587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:54.743558    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:56.372873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:56.372831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:56.912943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:56.912909    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:57.424726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.424689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:57.683729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.683603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:57.958552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.958471    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"]
Nov 02 18:05:57.959063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959047    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy"
Nov 02 18:05:57.959112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959065    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy"
Nov 02 18:05:57.959112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959079    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy"
Nov 02 18:05:57.959112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959084    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy"
Nov 02 18:05:57.959112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959097    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c8340cef-b8e3-435e-9bfd-e2b87e67420d" containerName="revoke-keys"
Nov 02 18:05:57.959112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959101    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8340cef-b8e3-435e-9bfd-e2b87e67420d" containerName="revoke-keys"
Nov 02 18:05:57.959315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959192    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy"
Nov 02 18:05:57.959315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959204    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7247c4-37c2-4e47-bb04-fa7d7d96b9eb" containerName="istio-proxy"
Nov 02 18:05:57.959315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.959214    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8340cef-b8e3-435e-9bfd-e2b87e67420d" containerName="revoke-keys"
Nov 02 18:05:57.963977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.963949    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:57.998103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.998073    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7kk\" (UniqueName: \"kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk\") pod \"payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:57.998252 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:57.998135    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:58.017063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.017035    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-948451\""
Nov 02 18:05:58.066207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.066168    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-948451-dockercfg-zs74t\""
Nov 02 18:05:58.070296 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.070268    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"]
Nov 02 18:05:58.099303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.099272    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7kk\" (UniqueName: \"kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk\") pod \"payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:58.099468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.099334    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:58.100108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.100084    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:58.163349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.163311    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7kk\" (UniqueName: \"kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk\") pod \"payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:58.279201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.279114    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:58.662835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.662774    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"]
Nov 02 18:05:58.664194 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:05:58.664166    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8deb8b_40bc_43de_ba74_a0ab298a5695.slice/crio-53d2f3d991d3b91b4bb970ae80d69d8d5db74823d2128e11e9911cfcab8a4f4c WatchSource:0}: Error finding container 53d2f3d991d3b91b4bb970ae80d69d8d5db74823d2128e11e9911cfcab8a4f4c: Status 404 returned error can't find the container with id 53d2f3d991d3b91b4bb970ae80d69d8d5db74823d2128e11e9911cfcab8a4f4c
Nov 02 18:05:58.696680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.696654    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9" event={"ID":"bd8deb8b-40bc-43de-ba74-a0ab298a5695","Type":"ContainerStarted","Data":"53d2f3d991d3b91b4bb970ae80d69d8d5db74823d2128e11e9911cfcab8a4f4c"}
Nov 02 18:05:58.701057 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.701036    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"]
Nov 02 18:05:58.706073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.706055    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:58.771999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.771964    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"]
Nov 02 18:05:58.807838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.807806    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvll7\" (UniqueName: \"kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7\") pod \"payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:58.808041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.807848    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:58.908793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.908751    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvll7\" (UniqueName: \"kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7\") pod \"payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:58.908793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.908796    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:58.909563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.909541    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:58.972716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:58.972619    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvll7\" (UniqueName: \"kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7\") pod \"payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:59.021770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.021743    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:59.062237 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:05:59.062196    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:05:59.404842 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.404816    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"]
Nov 02 18:05:59.405708 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:05:59.405684    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f34649d_98bd_48f2_a562_dd21caff8a55.slice/crio-34592ed2f8c71698c7b8c1113c04aced5c114f4445d762912f14bd74665b5b81 WatchSource:0}: Error finding container 34592ed2f8c71698c7b8c1113c04aced5c114f4445d762912f14bd74665b5b81: Status 404 returned error can't find the container with id 34592ed2f8c71698c7b8c1113c04aced5c114f4445d762912f14bd74665b5b81
Nov 02 18:05:59.705180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.705075    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb" event={"ID":"6f34649d-98bd-48f2-a562-dd21caff8a55","Type":"ContainerStarted","Data":"e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc"}
Nov 02 18:05:59.705660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.705194    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:05:59.705660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.705246    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb" event={"ID":"6f34649d-98bd-48f2-a562-dd21caff8a55","Type":"ContainerStarted","Data":"34592ed2f8c71698c7b8c1113c04aced5c114f4445d762912f14bd74665b5b81"}
Nov 02 18:05:59.706964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.706934    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9" event={"ID":"bd8deb8b-40bc-43de-ba74-a0ab298a5695","Type":"ContainerStarted","Data":"297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa"}
Nov 02 18:05:59.707093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.707038    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:05:59.791187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.791155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:59.823899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.823849    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb" podStartSLOduration=1.8238360249999999 podStartE2EDuration="1.823836025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:59.820545034 +0000 UTC m=+2590.661150967" watchObservedRunningTime="2026-11-02 18:05:59.823836025 +0000 UTC m=+2590.664441953"
Nov 02 18:05:59.937794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:05:59.937715    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9" podStartSLOduration=2.9376981669999997 podStartE2EDuration="2.937698167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:59.93572683 +0000 UTC m=+2590.776332765" watchObservedRunningTime="2026-11-02 18:05:59.937698167 +0000 UTC m=+2590.778304102"
Nov 02 18:06:03.221371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.221336    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"]
Nov 02 18:06:03.225416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.225397    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.277031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.276989    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6e3175-openshift-default-dockercfg-n7khr\""
Nov 02 18:06:03.331350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.331317    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"]
Nov 02 18:06:03.360689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360653    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.360876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360711    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.360876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360751    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.360876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360842    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.361046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360912    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.361046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360940    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.361046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.360969    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.361046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.361011    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjrt\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.361046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.361035    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461738    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461783    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461808    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461853    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461891    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461918    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.461958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461944    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.462267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.461988    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjrt\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.462267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.462014    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.462267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.462229    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.462450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.462419    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.462587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.462565    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.462910 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.462869    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.463032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.463007    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.464459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.464429    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.464681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.464660    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.519466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.519404    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.519672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.519648    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjrt\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.539600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.539575    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:03.978607 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.978569    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"]
Nov 02 18:06:03.979440 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:06:03.979413    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe920c1e_eb2a_4e61_bdee_52150339ef05.slice/crio-dcbcff7d83f79cdf6728f85c7de8eb35eaa3c5c94f7e9bb4395451c32825d5e6 WatchSource:0}: Error finding container dcbcff7d83f79cdf6728f85c7de8eb35eaa3c5c94f7e9bb4395451c32825d5e6: Status 404 returned error can't find the container with id dcbcff7d83f79cdf6728f85c7de8eb35eaa3c5c94f7e9bb4395451c32825d5e6
Nov 02 18:06:03.982113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.982077    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:06:03.982209 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.982152    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:06:03.982209 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:03.982186    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:06:04.706979 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:04.706938    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:04.747647 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:04.747588    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" event={"ID":"be920c1e-eb2a-4e61-bdee-52150339ef05","Type":"ContainerStarted","Data":"3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516"}
Nov 02 18:06:04.747951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:04.747919    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" event={"ID":"be920c1e-eb2a-4e61-bdee-52150339ef05","Type":"ContainerStarted","Data":"dcbcff7d83f79cdf6728f85c7de8eb35eaa3c5c94f7e9bb4395451c32825d5e6"}
Nov 02 18:06:04.876985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:04.876911    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podStartSLOduration=1.8768884529999998 podStartE2EDuration="1.876888453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:04.866894195 +0000 UTC m=+2595.707500133" watchObservedRunningTime="2026-11-02 18:06:04.876888453 +0000 UTC m=+2595.717494388"
Nov 02 18:06:05.550156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:05.550124    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:05.553757 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:05.553723    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 02 18:06:05.553994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:05.553967    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 02 18:06:06.561653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:06.561011    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 02 18:06:06.561653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:06.561079    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 02 18:06:07.549654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:07.545959    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 02 18:06:07.549654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:07.546016    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 02 18:06:08.563655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:08.560942    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 02 18:06:08.563655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:08.561004    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 02 18:06:09.542606 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:09.542571    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 02 18:06:09.542933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:09.542904    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 02 18:06:10.541271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:10.541190    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 02 18:06:10.541271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:10.541262    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 02 18:06:10.716559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:10.716522    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:06:10.717505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:10.717477    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:06:10.771226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:10.771171    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:11.052521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:11.052484    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:11.314580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:11.314489    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:12.644654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:12.643715    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:06:12.644654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:12.643790    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:06:13.685831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:13.685779    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:06:13.686335 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:13.685849    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:06:14.047849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:14.047759    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:14.048688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:14.048662    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:14.049271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:14.049252    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:06:15.567811 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:15.567773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:17.598163 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:17.598132    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:17.831573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:17.831533    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:18.308295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:18.308257    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb_6f34649d-98bd-48f2-a562-dd21caff8a55/payload-processing/0.log"
Nov 02 18:06:19.554558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:19.554528    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:20.846889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:20.846859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:21.101311 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:21.101187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:22.765023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:22.764993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:23.216049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.216013    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:23.529135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.529042    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"]
Nov 02 18:06:23.533356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.533331    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.589860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.589815    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.589887    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590087    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590168    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590219    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590254    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590316    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590481    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxgw\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.590573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.590525    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.646378 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.646334    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"]
Nov 02 18:06:23.692066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692034    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692072    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692137    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692166    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692188    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692288    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxgw\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692328    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692368    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692429    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.692937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.692913    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.693042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.693006    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.693342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.693320    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.693425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.693387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.693777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.693752    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.695211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.695187    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.696558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.696536    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.752657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.751270    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.765655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.763856    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxgw\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:23.848133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:23.848029    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:24.164903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.164865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:24.299496 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.299454    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"]
Nov 02 18:06:24.300269 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:06:24.300237    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71345efa_c26a_4e14_a8ad_fb745aaa9eef.slice/crio-34276c35ff371a4b0f30b85fbb995491a25bcc1f26f404f7881b6c92821fd617 WatchSource:0}: Error finding container 34276c35ff371a4b0f30b85fbb995491a25bcc1f26f404f7881b6c92821fd617: Status 404 returned error can't find the container with id 34276c35ff371a4b0f30b85fbb995491a25bcc1f26f404f7881b6c92821fd617
Nov 02 18:06:24.303245 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.303203    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:06:24.303355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.303289    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:06:24.303355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.303335    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:06:24.380472 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.380440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:24.582956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.582857    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:24.956825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.956777    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" event={"ID":"71345efa-c26a-4e14-a8ad-fb745aaa9eef","Type":"ContainerStarted","Data":"80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116"}
Nov 02 18:06:24.957358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:24.956853    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" event={"ID":"71345efa-c26a-4e14-a8ad-fb745aaa9eef","Type":"ContainerStarted","Data":"34276c35ff371a4b0f30b85fbb995491a25bcc1f26f404f7881b6c92821fd617"}
Nov 02 18:06:25.133222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:25.133151    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podStartSLOduration=2.13313158 podStartE2EDuration="2.13313158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:25.128482828 +0000 UTC m=+2615.969088763" watchObservedRunningTime="2026-11-02 18:06:25.13313158 +0000 UTC m=+2615.973737516"
Nov 02 18:06:25.511193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:25.509763    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:25.851029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:25.850932    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 18:06:25.851288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:25.851257    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 18:06:25.851694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:25.851672    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:25.977364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:25.977314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:26.182294 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:26.182262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:26.849196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:26.849160    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 18:06:26.849534 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:26.849483    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 18:06:27.855655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:27.851042    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 18:06:27.855655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:27.851105    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 18:06:28.164132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:28.164084    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:28.849060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:28.849020    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 18:06:28.849365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:28.849332    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 18:06:29.008559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:29.008524    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:29.094970 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:06:29.094934    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:06:29.347654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:29.344493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:29.854953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:29.854036    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 18:06:29.854953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:29.854103    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 18:06:29.918284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:29.918250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:30.173857 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:30.173828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:30.392853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:30.392819    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:30.848474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:30.848434    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 18:06:30.848674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:30.848500    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 18:06:31.853609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:31.853577    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:31.854282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:31.854263    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:31.854979 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:31.854961    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:06:32.326507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.326477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:32.583294 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.583218    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:32.664568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.664532    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"]
Nov 02 18:06:32.669892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.669870    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.722157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.722128    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f7aa2e-dockercfg-qz6np\""
Nov 02 18:06:32.722373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.722352    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f7aa2e\""
Nov 02 18:06:32.725509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.725487    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.725667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.725527    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxxt\" (UniqueName: \"kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt\") pod \"payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.726114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.726095    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"]
Nov 02 18:06:32.827250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.827214    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.827478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.827277    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxxt\" (UniqueName: \"kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt\") pod \"payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.828136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.828109    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.884651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.884595    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxxt\" (UniqueName: \"kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt\") pod \"payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:32.984819 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:32.984786    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:33.365054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:33.365024    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"]
Nov 02 18:06:33.365834 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:06:33.365757    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3473e48_f176_45c0_a801_da43ccf16d65.slice/crio-fffdf9ed753cd133fb97bb19fdf1c25609664abb99d18d3265d822accd5fa6fd WatchSource:0}: Error finding container fffdf9ed753cd133fb97bb19fdf1c25609664abb99d18d3265d822accd5fa6fd: Status 404 returned error can't find the container with id fffdf9ed753cd133fb97bb19fdf1c25609664abb99d18d3265d822accd5fa6fd
Nov 02 18:06:33.877974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:33.877937    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:34.061077 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:34.061044    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" event={"ID":"f3473e48-f176-45c0-a801-da43ccf16d65","Type":"ContainerStarted","Data":"3bff335b0642215f9ac7e8b30fb324bfefd826d784f2b94bdd89bf15e91aa503"}
Nov 02 18:06:34.061584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:34.061095    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" event={"ID":"f3473e48-f176-45c0-a801-da43ccf16d65","Type":"ContainerStarted","Data":"fffdf9ed753cd133fb97bb19fdf1c25609664abb99d18d3265d822accd5fa6fd"}
Nov 02 18:06:34.061584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:34.061266    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:34.178529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:34.178426    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" podStartSLOduration=2.178411678 podStartE2EDuration="2.178411678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:34.177437357 +0000 UTC m=+2625.018043292" watchObservedRunningTime="2026-11-02 18:06:34.178411678 +0000 UTC m=+2625.019017611"
Nov 02 18:06:37.184447 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.184409    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:37.384751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.384715    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:37.572651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.572542    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"]
Nov 02 18:06:37.578461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.578437    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:37.622923 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.622890    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"]
Nov 02 18:06:37.678654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.668654    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-413a4e\""
Nov 02 18:06:37.678654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.669364    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-413a4e-dockercfg-smt8h\""
Nov 02 18:06:37.724968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.722511    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6l5z\" (UniqueName: \"kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z\") pod \"payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:37.724968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.722622    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:37.846715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.840676    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6l5z\" (UniqueName: \"kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z\") pod \"payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:37.846715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.840803    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:37.846715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.841952    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:37.925711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:37.925587    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6l5z\" (UniqueName: \"kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z\") pod \"payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:38.208892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:38.208853    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:38.899152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:38.898717    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"]
Nov 02 18:06:38.900136 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:06:38.900109    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001c983b_7313_493d_9fec_04b983993105.slice/crio-1da45afa4771c62516452babc9f98e5f8a2dd585ef4db23835d7933e66836b22 WatchSource:0}: Error finding container 1da45afa4771c62516452babc9f98e5f8a2dd585ef4db23835d7933e66836b22: Status 404 returned error can't find the container with id 1da45afa4771c62516452babc9f98e5f8a2dd585ef4db23835d7933e66836b22
Nov 02 18:06:39.130465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:39.130424    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" event={"ID":"001c983b-7313-493d-9fec-04b983993105","Type":"ContainerStarted","Data":"c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4"}
Nov 02 18:06:39.130688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:39.130488    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" event={"ID":"001c983b-7313-493d-9fec-04b983993105","Type":"ContainerStarted","Data":"1da45afa4771c62516452babc9f98e5f8a2dd585ef4db23835d7933e66836b22"}
Nov 02 18:06:39.130748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:39.130691    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:39.255016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:39.254905    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" podStartSLOduration=2.254873711 podStartE2EDuration="2.254873711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:39.249967629 +0000 UTC m=+2630.090573565" watchObservedRunningTime="2026-11-02 18:06:39.254873711 +0000 UTC m=+2630.095479646"
Nov 02 18:06:40.141150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:40.141087    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:45.074208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:45.074179    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:06:46.919173 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:46.919140    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:47.110109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:47.110072    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:50.143329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:50.143293    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:06:51.792787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:51.792744    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:54.775396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:54.775363    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:56.667348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:06:56.667313    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:59.060065 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:06:59.060024    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:07:01.369673 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:01.369540    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:03.675884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.675848    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"]
Nov 02 18:07:03.681268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.681239    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.725776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.725735    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"]
Nov 02 18:07:03.730873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.730844    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.781831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.781794    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"]
Nov 02 18:07:03.782509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782482    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.782649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782525    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.782717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782686    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.782779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782718    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.782779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782747    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.782888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782789    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.782943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782922    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.782998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782951    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.782998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782969    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.783104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.782998    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.783104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783077    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.783214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783112    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.783214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783163    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6skg\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.783214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783188    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.783374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783227    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqt9r\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.783374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783269    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.783374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783316    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.783374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.783363    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.830216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.830170    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"]
Nov 02 18:07:03.884981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.884948    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.884981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.884987    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885008    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6skg\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885031    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885070    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqt9r\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885111    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885158    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885207    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885253    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885275    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885338    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885369    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885394    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885417    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885436    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885512    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885547    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.885583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885573    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885613    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.885785    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886261 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886165    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.886261 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886201    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.886261 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886207    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886417    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886467    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886492    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.886602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886495    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.886876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.886686    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.887888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.887854    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.888054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.888032    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.888153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.888136    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.888644 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.888607    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.942601 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.942527    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqt9r\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.943028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.943007    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:03.943103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.943055    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.943192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.943174    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6skg\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:03.997375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:03.997343    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:04.047398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.047346    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:04.436885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.436820    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"]
Nov 02 18:07:04.437539 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:07:04.437511    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2232ebee_9107_4f3d_b762_962d009dc173.slice/crio-2cacd41a01295ce109956432a7c7bb23cf755e77c8d30ac1f5a3ee97515c08f2 WatchSource:0}: Error finding container 2cacd41a01295ce109956432a7c7bb23cf755e77c8d30ac1f5a3ee97515c08f2: Status 404 returned error can't find the container with id 2cacd41a01295ce109956432a7c7bb23cf755e77c8d30ac1f5a3ee97515c08f2
Nov 02 18:07:04.440170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.440135    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:04.440264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.440210    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:04.440330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.440270    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:04.489743 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.489705    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"]
Nov 02 18:07:04.492744 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:07:04.492692    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269da4e5_61d1_4305_89b5_c52443fc6815.slice/crio-acc80425b6cd212d69f87e2edc37cf495412ca4a6facb4f7320477fc17fed3bb WatchSource:0}: Error finding container acc80425b6cd212d69f87e2edc37cf495412ca4a6facb4f7320477fc17fed3bb: Status 404 returned error can't find the container with id acc80425b6cd212d69f87e2edc37cf495412ca4a6facb4f7320477fc17fed3bb
Nov 02 18:07:04.495887 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.495849    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:04.495972 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.495949    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:04.496018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:04.495989    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:05.348655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.333953    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" event={"ID":"269da4e5-61d1-4305-89b5-c52443fc6815","Type":"ContainerStarted","Data":"1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1"}
Nov 02 18:07:05.348655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.334064    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" event={"ID":"269da4e5-61d1-4305-89b5-c52443fc6815","Type":"ContainerStarted","Data":"acc80425b6cd212d69f87e2edc37cf495412ca4a6facb4f7320477fc17fed3bb"}
Nov 02 18:07:05.348655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.336151    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" event={"ID":"2232ebee-9107-4f3d-b762-962d009dc173","Type":"ContainerStarted","Data":"6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30"}
Nov 02 18:07:05.348655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.336198    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" event={"ID":"2232ebee-9107-4f3d-b762-962d009dc173","Type":"ContainerStarted","Data":"2cacd41a01295ce109956432a7c7bb23cf755e77c8d30ac1f5a3ee97515c08f2"}
Nov 02 18:07:05.524352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.524275    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podStartSLOduration=2.524254832 podStartE2EDuration="2.524254832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:05.520440477 +0000 UTC m=+2656.361046412" watchObservedRunningTime="2026-11-02 18:07:05.524254832 +0000 UTC m=+2656.364860767"
Nov 02 18:07:05.560822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.553602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:05.716681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.715201    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podStartSLOduration=2.715180108 podStartE2EDuration="2.715180108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:05.692511198 +0000 UTC m=+2656.533117132" watchObservedRunningTime="2026-11-02 18:07:05.715180108 +0000 UTC m=+2656.555786038"
Nov 02 18:07:06.001783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:05.999806    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:06.002168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.002145    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:07:06.008115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.008071    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 18:07:06.048071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.048034    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:06.050786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.050753    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 18:07:06.051055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.051028    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 18:07:06.093367 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.093279    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:06.299942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.299835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:06.999076 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.999029    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:07:06.999600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:06.999101    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 18:07:07.064654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.060993    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 18:07:07.064654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.061049    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 18:07:07.141281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.138391    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:07.387649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.386970    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:07.632297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.632263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:07.819573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.819494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:07.999680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.998001    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:07:07.999680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:07.998060    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 18:07:08.050654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:08.048981    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 18:07:08.050654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:08.049056    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 18:07:09.001001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:09.000234    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:07:09.001001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:09.000313    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 18:07:09.029211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:09.029178    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:09.064653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:09.055200    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 18:07:09.064653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:09.055276    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 18:07:09.661651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:09.658177    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:10.005711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.004752    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:07:10.005711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.004816    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 18:07:10.050062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.050023    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 18:07:10.050342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.050310    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 18:07:10.374231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.374151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:10.998002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.997965    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 02 18:07:10.998216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:10.998030    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 02 18:07:11.048939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:11.048892    4476 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 02 18:07:11.049468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:11.048965    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 02 18:07:12.422287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.422253    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:12.422845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.422834    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:12.423707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.423683    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:07:12.546430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.546167    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:12.546821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.546799    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:12.547778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.547761    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:07:12.879352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:12.879326    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:16.516201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:16.516161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:17.952157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:17.952120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:18.155433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:18.155395    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:18.373531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:18.373503    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:23.181310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:23.181278    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:27.823011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:27.822976    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:28.024529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:28.024490    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:28.243034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:28.243001    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:29.061587 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:07:29.061553    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:07:33.074000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:33.073956    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:37.417548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:37.417513    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:37.758283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:37.758210    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"]
Nov 02 18:07:37.758505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:37.758480    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb" podUID="6f34649d-98bd-48f2-a562-dd21caff8a55" containerName="payload-processing" containerID="cri-o://e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc" gracePeriod=30
Nov 02 18:07:37.813329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:37.813297    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"]
Nov 02 18:07:37.813537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:37.813501    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9" podUID="bd8deb8b-40bc-43de-ba74-a0ab298a5695" containerName="payload-pre-processing" containerID="cri-o://297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa" gracePeriod=30
Nov 02 18:07:37.978359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:37.978315    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:38.104267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.104227    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:07:38.130704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.130681    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:07:38.140585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.140561    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume\") pod \"6f34649d-98bd-48f2-a562-dd21caff8a55\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") "
Nov 02 18:07:38.140801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.140786    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7\" (UniqueName: \"kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7\") pod \"6f34649d-98bd-48f2-a562-dd21caff8a55\" (UID: \"6f34649d-98bd-48f2-a562-dd21caff8a55\") "
Nov 02 18:07:38.140896 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.140875    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume" pod "6f34649d-98bd-48f2-a562-dd21caff8a55" (UID: "6f34649d-98bd-48f2-a562-dd21caff8a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:38.141105 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.141091    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f34649d-98bd-48f2-a562-dd21caff8a55-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:38.143056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.143027    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7" pod "6f34649d-98bd-48f2-a562-dd21caff8a55" (UID: "6f34649d-98bd-48f2-a562-dd21caff8a55"). InnerVolumeSpecName "kube-api-access-wvll7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:38.210991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.210965    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:38.241933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.241894    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk\" (UniqueName: \"kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk\") pod \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") "
Nov 02 18:07:38.242111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.241952    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume\") pod \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\" (UID: \"bd8deb8b-40bc-43de-ba74-a0ab298a5695\") "
Nov 02 18:07:38.242463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.242433    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume" pod "bd8deb8b-40bc-43de-ba74-a0ab298a5695" (UID: "bd8deb8b-40bc-43de-ba74-a0ab298a5695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:38.242619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.242581    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd8deb8b-40bc-43de-ba74-a0ab298a5695-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:38.242619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.242604    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvll7\" (UniqueName: \"kubernetes.io/projected/6f34649d-98bd-48f2-a562-dd21caff8a55-kube-api-access-wvll7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:38.244882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.244855    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk" pod "bd8deb8b-40bc-43de-ba74-a0ab298a5695" (UID: "bd8deb8b-40bc-43de-ba74-a0ab298a5695"). InnerVolumeSpecName "kube-api-access-pf7kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:38.358901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.358821    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf7kk\" (UniqueName: \"kubernetes.io/projected/bd8deb8b-40bc-43de-ba74-a0ab298a5695-kube-api-access-pf7kk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:07:38.716834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.716799    4476 generic.go:356] "Generic (PLEG): container finished" podID="bd8deb8b-40bc-43de-ba74-a0ab298a5695" containerID="297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa" exitCode=0
Nov 02 18:07:38.717280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.716853    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"
Nov 02 18:07:38.717280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.716888    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9" event={"ID":"bd8deb8b-40bc-43de-ba74-a0ab298a5695","Type":"ContainerDied","Data":"297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa"}
Nov 02 18:07:38.717280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.716941    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9" event={"ID":"bd8deb8b-40bc-43de-ba74-a0ab298a5695","Type":"ContainerDied","Data":"53d2f3d991d3b91b4bb970ae80d69d8d5db74823d2128e11e9911cfcab8a4f4c"}
Nov 02 18:07:38.717280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.716961    4476 scope.go:122] "RemoveContainer" containerID="297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa"
Nov 02 18:07:38.718576 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.718551    4476 generic.go:356] "Generic (PLEG): container finished" podID="6f34649d-98bd-48f2-a562-dd21caff8a55" containerID="e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc" exitCode=0
Nov 02 18:07:38.718716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.718637    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb" event={"ID":"6f34649d-98bd-48f2-a562-dd21caff8a55","Type":"ContainerDied","Data":"e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc"}
Nov 02 18:07:38.718716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.718671    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"
Nov 02 18:07:38.718716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.718691    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb" event={"ID":"6f34649d-98bd-48f2-a562-dd21caff8a55","Type":"ContainerDied","Data":"34592ed2f8c71698c7b8c1113c04aced5c114f4445d762912f14bd74665b5b81"}
Nov 02 18:07:38.728621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.728605    4476 scope.go:122] "RemoveContainer" containerID="297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa"
Nov 02 18:07:38.728915 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:07:38.728895    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa\": container with ID starting with 297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa not found: ID does not exist" containerID="297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa"
Nov 02 18:07:38.728992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.728923    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa"} err="failed to get container status \"297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa\": rpc error: code = NotFound desc = could not find container \"297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa\": container with ID starting with 297431d31b924c45429e23357667a270518e802fa371bc23f6cd0d227f79dfaa not found: ID does not exist"
Nov 02 18:07:38.728992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.728942    4476 scope.go:122] "RemoveContainer" containerID="e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc"
Nov 02 18:07:38.737811 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.737794    4476 scope.go:122] "RemoveContainer" containerID="e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc"
Nov 02 18:07:38.738077 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:07:38.738061    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc\": container with ID starting with e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc not found: ID does not exist" containerID="e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc"
Nov 02 18:07:38.738135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.738084    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc"} err="failed to get container status \"e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc\": rpc error: code = NotFound desc = could not find container \"e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc\": container with ID starting with e4cb76fc359e1759e457984146de04dd8ebaf6a417d5d9e1ffdb317a95ac87fc not found: ID does not exist"
Nov 02 18:07:38.898671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.898621    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"]
Nov 02 18:07:38.948809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:38.948780    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-b78dc67dc-ffsj9"]
Nov 02 18:07:39.015027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.014955    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"]
Nov 02 18:07:39.020716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.020690    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-7bd4cbf79d-l5dcb"]
Nov 02 18:07:39.049675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.049643    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"]
Nov 02 18:07:39.053241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.053216    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc07ab0f-8p7vs"]
Nov 02 18:07:39.315163 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.315081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:39.384711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.384671    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5d28d1-30e7-4254-a850-bf6aafb4df25" path="/var/lib/kubelet/pods/1d5d28d1-30e7-4254-a850-bf6aafb4df25/volumes"
Nov 02 18:07:39.385223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.385204    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f34649d-98bd-48f2-a562-dd21caff8a55" path="/var/lib/kubelet/pods/6f34649d-98bd-48f2-a562-dd21caff8a55/volumes"
Nov 02 18:07:39.385758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:39.385740    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8deb8b-40bc-43de-ba74-a0ab298a5695" path="/var/lib/kubelet/pods/bd8deb8b-40bc-43de-ba74-a0ab298a5695/volumes"
Nov 02 18:07:40.628122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:40.628083    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:40.879334 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:40.879249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:42.253402 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:42.253373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:42.759434 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:42.759392    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:44.250219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:44.250189    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:44.719407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:44.719364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:44.932026 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:44.931984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:45.193154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:45.193121    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:45.981146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:45.981101    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:46.401253 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.401219    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:46.899117 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.899080    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"]
Nov 02 18:07:46.900094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.900065    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bd8deb8b-40bc-43de-ba74-a0ab298a5695" containerName="payload-pre-processing"
Nov 02 18:07:46.900094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.900093    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8deb8b-40bc-43de-ba74-a0ab298a5695" containerName="payload-pre-processing"
Nov 02 18:07:46.900305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.900112    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6f34649d-98bd-48f2-a562-dd21caff8a55" containerName="payload-processing"
Nov 02 18:07:46.900305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.900121    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f34649d-98bd-48f2-a562-dd21caff8a55" containerName="payload-processing"
Nov 02 18:07:46.900305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.900266    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8deb8b-40bc-43de-ba74-a0ab298a5695" containerName="payload-pre-processing"
Nov 02 18:07:46.900305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.900279    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f34649d-98bd-48f2-a562-dd21caff8a55" containerName="payload-processing"
Nov 02 18:07:46.903996 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.903977    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.956740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.956710    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-4ecad8-openshift-default-dockercfg-c9xjq\""
Nov 02 18:07:46.971468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.971420    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vz8\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.971851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.971827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.972069 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.972052    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.972182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.972169    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.972283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.972269    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.974654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.972382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.974885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.974864    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.975089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.975056    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:46.975227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:46.975092    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.024862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.024822    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"]
Nov 02 18:07:47.066524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.066430    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:47.072753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.072726    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"]
Nov 02 18:07:47.076614 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076580    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.076786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076688    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.076786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076728    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.076786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076756    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076794    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076842    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076879    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.076910    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.077053    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vz8\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.077222    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d2809f-zwp8q"]
Nov 02 18:07:47.077546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.077419    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.077539    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.077817    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.077980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.077962    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.078060 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.078042    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.080262 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.080238    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.080501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.080485    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.136129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.136092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.136315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.136160    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vz8\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8\") pod \"e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") " pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.217318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.217228    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:47.286380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.286265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:47.383241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.383205    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33668b28-3708-425e-80b2-06f95a98305c" path="/var/lib/kubelet/pods/33668b28-3708-425e-80b2-06f95a98305c/volumes"
Nov 02 18:07:47.661288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.661251    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"]
Nov 02 18:07:47.661889 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:07:47.661864    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a85290_1c16_4c6b_8cfe_89922ebac737.slice/crio-40e593c9d4e4b828cfc717a588740fbea9399d8402aff6ecc151a3ace503e7ac WatchSource:0}: Error finding container 40e593c9d4e4b828cfc717a588740fbea9399d8402aff6ecc151a3ace503e7ac: Status 404 returned error can't find the container with id 40e593c9d4e4b828cfc717a588740fbea9399d8402aff6ecc151a3ace503e7ac
Nov 02 18:07:47.664640 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.664593    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:47.664746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.664718    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:47.664817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.664764    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:07:47.811308 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.811274    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" event={"ID":"03a85290-1c16-4c6b-8cfe-89922ebac737","Type":"ContainerStarted","Data":"b763fac65251504cace4a9113ca2c45787cf534071dac5e029e6e0ea37cf3151"}
Nov 02 18:07:47.811557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.811340    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" event={"ID":"03a85290-1c16-4c6b-8cfe-89922ebac737","Type":"ContainerStarted","Data":"40e593c9d4e4b828cfc717a588740fbea9399d8402aff6ecc151a3ace503e7ac"}
Nov 02 18:07:47.929667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:47.929506    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podStartSLOduration=1.9294906840000001 podStartE2EDuration="1.929490684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:47.927979021 +0000 UTC m=+2698.768584961" watchObservedRunningTime="2026-11-02 18:07:47.929490684 +0000 UTC m=+2698.770096618"
Nov 02 18:07:48.218418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:48.218323    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:48.224534 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:48.224504    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:48.818660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:48.818601    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:48.819574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:48.819545    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:07:49.802910 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:49.802877    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:07:49.811689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:49.811662    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:07:50.004362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:50.004327    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:50.228683 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:50.228620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:50.445965 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:50.445927    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:50.767342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:50.767311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:51.212690 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:51.212654    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:51.444062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:51.444028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:51.701647 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:51.701593    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:53.417540 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:53.417512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:53.606847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:53.606816    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:55.110548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:55.110516    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:56.652372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:56.652332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:58.407700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:58.407661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:58.603526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:07:58.603494    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:07:59.059747 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:07:59.059709    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:08:00.481185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:00.481151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:05.047872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:05.047830    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"]
Nov 02 18:08:05.096371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:05.096336    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7c3511-tb9bk"]
Nov 02 18:08:05.386920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:05.386879    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65aa76e-6384-45f9-8b7a-97a919742be6" path="/var/lib/kubelet/pods/b65aa76e-6384-45f9-8b7a-97a919742be6/volumes"
Nov 02 18:08:05.511243 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:05.511208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:09.763350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:09.763318    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:10.216386 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:10.216350    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:10.682731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:10.682701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:10.889702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:10.889669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:11.083268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:11.083181    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:15.759769 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:15.759619    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:16.085333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:16.085251    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:16.290114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:16.290080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:20.990249 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:20.990210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:21.195528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:21.195498    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:21.387509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:21.387478    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:21.580159 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:21.580121    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:21.997939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:21.997905    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"]
Nov 02 18:08:22.002361 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.002337    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.076931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.076899    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2a1bcc-openshift-default-dockercfg-94jzs\""
Nov 02 18:08:22.092251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.092220    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"]
Nov 02 18:08:22.109674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109342    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.109674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109394    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.109674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109450    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.109674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109489    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnb6d\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.109674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109535    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.109674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109614    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.110133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109771    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.110133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109797    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.110133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.109820    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.211738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.211677    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.212093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.212069    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnb6d\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.212310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.212294    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.212554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.212538    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.212808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.212788    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.212979 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.212962    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.213128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.213111    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.213391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.213374    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.213573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.213543    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.214257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.214228    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.218190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.215097    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.218190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.215587    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.218190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.216376    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.218190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.217221    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.227762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.218237    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.234111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.234040    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.281031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.280937    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.281247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.281224    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnb6d\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d\") pod \"e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") " pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.283415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.283385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:22.316724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.316693    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:22.506384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.506322    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:22.766030 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:08:22.765993    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5269bdd_c304_425e_ae66_f6d1b2242bb3.slice/crio-e0b93509a6860e50daa3bdf564875324f9f25519a77938bb4e8cfc9dc8c8a87e WatchSource:0}: Error finding container e0b93509a6860e50daa3bdf564875324f9f25519a77938bb4e8cfc9dc8c8a87e: Status 404 returned error can't find the container with id e0b93509a6860e50daa3bdf564875324f9f25519a77938bb4e8cfc9dc8c8a87e
Nov 02 18:08:22.767708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.767678    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"]
Nov 02 18:08:22.768801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.768771    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:08:22.768913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.768830    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:08:22.768913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:22.768860    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:08:23.155813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:23.155768    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" event={"ID":"f5269bdd-c304-425e-ae66-f6d1b2242bb3","Type":"ContainerStarted","Data":"39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"}
Nov 02 18:08:23.156317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:23.155846    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" event={"ID":"f5269bdd-c304-425e-ae66-f6d1b2242bb3","Type":"ContainerStarted","Data":"e0b93509a6860e50daa3bdf564875324f9f25519a77938bb4e8cfc9dc8c8a87e"}
Nov 02 18:08:23.272913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:23.272853    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podStartSLOduration=2.272833638 podStartE2EDuration="2.272833638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:23.271145291 +0000 UTC m=+2734.111751239" watchObservedRunningTime="2026-11-02 18:08:23.272833638 +0000 UTC m=+2734.113439574"
Nov 02 18:08:23.317822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:23.317786    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:23.323877 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:23.323850    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:24.163778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:24.163746    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:24.164724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:24.164706    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:08:26.440583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:26.440553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:26.758704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:26.758585    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:26.952916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:26.952889    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:27.161340 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:27.161301    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:27.583097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:27.583022    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:28.149701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:28.149666    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:28.392096 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:28.392059    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:29.060301 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:08:29.060266    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:08:30.836354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:30.836262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:31.047125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:31.047090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:31.633429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:31.633393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:32.639308 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:32.639276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:32.881683 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:32.881647    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:35.914469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:35.914420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:36.100208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:36.100169    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:36.920330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:36.920262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:38.050136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:38.050102    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"]
Nov 02 18:08:38.099461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:38.099429    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2041fbc1-9b9gc"]
Nov 02 18:08:39.384107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:39.384066    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d6d1a8-cc1a-4130-8787-b6bbab8578e0" path="/var/lib/kubelet/pods/22d6d1a8-cc1a-4130-8787-b6bbab8578e0/volumes"
Nov 02 18:08:41.785752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.785716    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"]
Nov 02 18:08:41.790669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.790652    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.891759 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.891729    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-eb1633-serving-cert\""
Nov 02 18:08:41.895324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.895303    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjrl\" (UniqueName: \"kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.895394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.895373    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.895572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.895547    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.895656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.895591    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.896007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.895988    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"]
Nov 02 18:08:41.997197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.997165    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.997384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.997286    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.997384 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:08:41.997311    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-serving-cert: secret "maas-api-e2e-praxis-a-eb1633-serving-cert" not found
Nov 02 18:08:41.997384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.997314    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.997384 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:08:41.997378    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls podName:eb16be8a-51ff-42c5-81fe-03cd76da5890 nodeName:}" failed. No retries permitted until 2026-11-02 18:08:42.497357561 +0000 UTC m=+2753.337963478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls") pod "maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" (UID: "eb16be8a-51ff-42c5-81fe-03cd76da5890") : secret "maas-api-e2e-praxis-a-eb1633-serving-cert" not found
Nov 02 18:08:41.997610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.997446    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjrl\" (UniqueName: \"kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.998001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.997981    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:41.999979 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:41.999956    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:42.015561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.015450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:42.054954 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.054831    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjrl\" (UniqueName: \"kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:42.215573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.215540    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:42.403617 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.403581    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:42.503656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.503603    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:42.506310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.506286    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:42.603157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.603123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:42.705029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:42.704951    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:43.082525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:43.082494    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"]
Nov 02 18:08:43.083456 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:08:43.083432    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb16be8a_51ff_42c5_81fe_03cd76da5890.slice/crio-bca543e49e879af6c5d58a9d4204854c1903028741595fc3c6b46408a3a996b1 WatchSource:0}: Error finding container bca543e49e879af6c5d58a9d4204854c1903028741595fc3c6b46408a3a996b1: Status 404 returned error can't find the container with id bca543e49e879af6c5d58a9d4204854c1903028741595fc3c6b46408a3a996b1
Nov 02 18:08:43.084803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:43.084788    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:08:43.311140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:43.311107    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" event={"ID":"eb16be8a-51ff-42c5-81fe-03cd76da5890","Type":"ContainerStarted","Data":"bca543e49e879af6c5d58a9d4204854c1903028741595fc3c6b46408a3a996b1"}
Nov 02 18:08:44.320258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:44.320151    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" event={"ID":"eb16be8a-51ff-42c5-81fe-03cd76da5890","Type":"ContainerStarted","Data":"e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997"}
Nov 02 18:08:44.320258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:44.320222    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:44.441697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:44.441643    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" podStartSLOduration=2.562153697 podStartE2EDuration="3.441610935s" totalImagesPullingTime="879.457238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:44.437594267 +0000 UTC m=+2755.278200203" watchObservedRunningTime="2026-11-02 18:08:44.441610935 +0000 UTC m=+2755.282216870"
Nov 02 18:08:47.183661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.183613    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"]
Nov 02 18:08:47.187662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.187640    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.261803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.261771    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.262004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.261920    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.262004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.261996    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.262144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.262102    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.289783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.289745    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-4ecad8-serving-cert\""
Nov 02 18:08:47.293760 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.293734    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"]
Nov 02 18:08:47.363208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.363178    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.363382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.363278    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.363382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.363346    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.363496 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.363409    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.363554 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:08:47.363491    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-serving-cert: secret "maas-api-e2e-shared-b-w1-4ecad8-serving-cert" not found
Nov 02 18:08:47.363620 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:08:47.363577    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls podName:c6ad59c1-566a-4f0f-99d4-5e3df926ca5a nodeName:}" failed. No retries permitted until 2026-11-02 18:08:47.863555653 +0000 UTC m=+2758.704161589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls") pod "maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" (UID: "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a") : secret "maas-api-e2e-shared-b-w1-4ecad8-serving-cert" not found
Nov 02 18:08:47.364102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.364082    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.366277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.366252    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.419257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.419227    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:47.421492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.421455    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.636558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.636505    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:47.832341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.832303    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:47.871580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.871543    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:47.874394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:47.874363    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:48.049925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:48.049838    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"]
Nov 02 18:08:48.055809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:48.055785    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059e1cf5-b5w2t"]
Nov 02 18:08:48.101802 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:48.101767    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:48.533050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:48.533018    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"]
Nov 02 18:08:48.533421 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:08:48.533107    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ad59c1_566a_4f0f_99d4_5e3df926ca5a.slice/crio-e814a48b55268ffddca1aa55ae4cb0f8c6265843880958c956cdd315f43b580a WatchSource:0}: Error finding container e814a48b55268ffddca1aa55ae4cb0f8c6265843880958c956cdd315f43b580a: Status 404 returned error can't find the container with id e814a48b55268ffddca1aa55ae4cb0f8c6265843880958c956cdd315f43b580a
Nov 02 18:08:49.048255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:49.048221    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"]
Nov 02 18:08:49.097307 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:49.097273    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ea676c-g989d"]
Nov 02 18:08:49.367384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:49.367337    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" event={"ID":"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a","Type":"ContainerStarted","Data":"e814a48b55268ffddca1aa55ae4cb0f8c6265843880958c956cdd315f43b580a"}
Nov 02 18:08:49.386031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:49.385924    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d34613-cfef-410c-9924-2af18cdc655b" path="/var/lib/kubelet/pods/34d34613-cfef-410c-9924-2af18cdc655b/volumes"
Nov 02 18:08:49.386734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:49.386709    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90330102-9f35-4e13-a135-b22c26ad2b4f" path="/var/lib/kubelet/pods/90330102-9f35-4e13-a135-b22c26ad2b4f/volumes"
Nov 02 18:08:50.344473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:50.344427    4476 scope.go:122] "RemoveContainer" containerID="bfe3dcec413e2bab8dc44d922bacc34f9132a603fe3e09ed0dc562f9919dee31"
Nov 02 18:08:50.346099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:50.346072    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:08:50.368334 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:50.368311    4476 scope.go:122] "RemoveContainer" containerID="d8e0f5f29b1b2e2523388af1876bee92bf46a0888c21b53fb13a9c68329bd951"
Nov 02 18:08:50.380734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:50.380701    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" event={"ID":"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a","Type":"ContainerStarted","Data":"2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e"}
Nov 02 18:08:50.380956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:50.380771    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:50.583082 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:50.583029    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" podStartSLOduration=2.649282536 podStartE2EDuration="3.583009654s" totalImagesPullingTime="933.727118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:50.577237754 +0000 UTC m=+2761.417843690" watchObservedRunningTime="2026-11-02 18:08:50.583009654 +0000 UTC m=+2761.423615591"
Nov 02 18:08:52.047010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:52.046983    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"]
Nov 02 18:08:52.096654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:52.096612    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e34c24e-b79zs"]
Nov 02 18:08:52.495110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:52.495070    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:53.383655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:53.383606    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a0ac18-4849-4ac2-8e98-7d305a8a534f" path="/var/lib/kubelet/pods/b2a0ac18-4849-4ac2-8e98-7d305a8a534f/volumes"
Nov 02 18:08:55.441174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:55.441141    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:56.393489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:56.393461    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:08:57.287091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:57.287053    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:58.522141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:58.522102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:58.703680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:08:58.703621    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:08:59.060433 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:08:59.060401    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:09:01.960399 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:01.960359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:04.047243 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:04.047200    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"]
Nov 02 18:09:04.096202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:04.096166    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e36ef6-9mb2d"]
Nov 02 18:09:05.385645 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:05.385586    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2edd3e-3365-428d-8d24-a70dc5623451" path="/var/lib/kubelet/pods/3f2edd3e-3365-428d-8d24-a70dc5623451/volumes"
Nov 02 18:09:06.665328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:06.665265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:10.064747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:10.064703    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"]
Nov 02 18:09:10.109255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:10.109217    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0ee669b-skhbc"]
Nov 02 18:09:11.385490 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:11.385454    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6f0046-8f9b-4867-b403-0a997463f58d" path="/var/lib/kubelet/pods/1d6f0046-8f9b-4867-b403-0a997463f58d/volumes"
Nov 02 18:09:11.498447 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:11.498410    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:15.572351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:15.572322    4476 generic.go:356] "Generic (PLEG): container finished" podID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerID="b763fac65251504cace4a9113ca2c45787cf534071dac5e029e6e0ea37cf3151" exitCode=0
Nov 02 18:09:15.572793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:15.572408    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" event={"ID":"03a85290-1c16-4c6b-8cfe-89922ebac737","Type":"ContainerDied","Data":"b763fac65251504cace4a9113ca2c45787cf534071dac5e029e6e0ea37cf3151"}
Nov 02 18:09:15.573004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:15.572984    4476 scope.go:122] "RemoveContainer" containerID="b763fac65251504cace4a9113ca2c45787cf534071dac5e029e6e0ea37cf3151"
Nov 02 18:09:15.574948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:15.574912    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:09:15.575031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:15.575013    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:09:15.575079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:15.575065    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:09:16.121341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:16.121301    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:16.309583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:16.309554    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:16.495394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:16.495313    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:16.583372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:16.583333    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" event={"ID":"03a85290-1c16-4c6b-8cfe-89922ebac737","Type":"ContainerStarted","Data":"009ac9804e0a52d5f3d3679190158abb25478a7716507e8ed170f1b6aeff4fa4"}
Nov 02 18:09:17.218203 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:17.218169    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:09:17.219520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:17.219496    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:09:17.219583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:17.219561    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 18:09:18.217620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:18.217572    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:09:18.218170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:18.217667    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 18:09:18.819257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:18.819228    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:09:19.218164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:19.218137    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:09:19.218616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:19.218591    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 18:09:19.231498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:19.231475    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:19.460833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:19.460780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:20.218307 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:20.218268    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:09:20.218741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:20.218343    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 18:09:21.115994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:21.115952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:21.218528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:21.218489    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 18:09:21.218946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:21.218559    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 18:09:22.223122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:22.223094    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:09:22.223991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:22.223973    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:09:25.048770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:25.048735    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"]
Nov 02 18:09:25.098003 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:25.097968    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a35fbb9-cvqlf"]
Nov 02 18:09:25.384043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:25.384013    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d" path="/var/lib/kubelet/pods/09ab28ef-c8a1-490a-bcea-6be0b0dc9b1d/volumes"
Nov 02 18:09:25.970923 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:25.970889    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:29.059585 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:09:29.059550    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:09:30.548408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:30.548377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:30.731931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:30.731898    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:35.449475 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:35.449434    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:39.816086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.816050    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"]
Nov 02 18:09:39.819486 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.819469    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.849352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.849321    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.849498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.849430    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvlf\" (UniqueName: \"kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.849498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.849472    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.849589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.849516    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.872353 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.872329    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c82472-serving-cert\""
Nov 02 18:09:39.926020 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.925989    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"]
Nov 02 18:09:39.950445 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.950409    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvlf\" (UniqueName: \"kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.950657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.950471    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.950657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.950510    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.950759 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:09:39.950662    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-serving-cert: secret "maas-api-e2e-shared-b-w2-c82472-serving-cert" not found
Nov 02 18:09:39.950759 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.950726    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.950759 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:09:39.950739    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls podName:60d64f4b-7a32-4013-96dd-415fcd0d5e2f nodeName:}" failed. No retries permitted until 2026-11-02 18:09:40.450715953 +0000 UTC m=+2811.291321921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" (UID: "60d64f4b-7a32-4013-96dd-415fcd0d5e2f") : secret "maas-api-e2e-shared-b-w2-c82472-serving-cert" not found
Nov 02 18:09:39.951445 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.951414    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:39.957876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:39.957849    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:40.007695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:40.007662    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvlf\" (UniqueName: \"kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:40.071932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:40.071851    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:40.275817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:40.275785    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:40.457083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:40.457034    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:40.459996 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:40.459967    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:40.731448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:40.731361    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:41.124126 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:41.124091    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"]
Nov 02 18:09:41.125073 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:09:41.125048    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d64f4b_7a32_4013_96dd_415fcd0d5e2f.slice/crio-1bdd5ec9dad30b59916f54ad3dfb73e3d8a6aeb7584381751bad0b8c22a5a590 WatchSource:0}: Error finding container 1bdd5ec9dad30b59916f54ad3dfb73e3d8a6aeb7584381751bad0b8c22a5a590: Status 404 returned error can't find the container with id 1bdd5ec9dad30b59916f54ad3dfb73e3d8a6aeb7584381751bad0b8c22a5a590
Nov 02 18:09:41.794144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:41.794099    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" event={"ID":"60d64f4b-7a32-4013-96dd-415fcd0d5e2f","Type":"ContainerStarted","Data":"1bdd5ec9dad30b59916f54ad3dfb73e3d8a6aeb7584381751bad0b8c22a5a590"}
Nov 02 18:09:42.804706 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:42.804663    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" event={"ID":"60d64f4b-7a32-4013-96dd-415fcd0d5e2f","Type":"ContainerStarted","Data":"a0275eb17f4072d1db5dc60be5af1c33f9dbfd44d12d03e00e65295421a298cb"}
Nov 02 18:09:42.805233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:42.804770    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:42.924239 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:42.924171    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" podStartSLOduration=3.117482971 podStartE2EDuration="3.924149192s" totalImagesPullingTime="806.666221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:42.918907901 +0000 UTC m=+2813.759513836" watchObservedRunningTime="2026-11-02 18:09:42.924149192 +0000 UTC m=+2813.764755126"
Nov 02 18:09:45.115320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:45.115284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:48.817043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:48.817017    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:09:49.994124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:49.994065    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:50.410676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:50.410643    4476 scope.go:122] "RemoveContainer" containerID="57f9c7e74abb3c61e8a6043944da4c78fcbe190fa2f75405d96a4328114e67ae"
Nov 02 18:09:50.421191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:50.421162    4476 scope.go:122] "RemoveContainer" containerID="7e68cc6561ebafd1a9ca344379c479e7a5f5008675d90ff05c966915e0a93298"
Nov 02 18:09:50.430511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:50.430482    4476 scope.go:122] "RemoveContainer" containerID="a4f4f683549ac15a650d312b1a7fb8ea1ccecf9bfac4b93be2a7d5e5f71e55d1"
Nov 02 18:09:50.441580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:50.441559    4476 scope.go:122] "RemoveContainer" containerID="dd449cb6149af76e0666fcec0914e4537d9256b5f8ff2f7c0e17303c9c921546"
Nov 02 18:09:50.452613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:50.452586    4476 scope.go:122] "RemoveContainer" containerID="3ac41bc5792aaf040f17c62a59dbe98a71e63ca1668f566f9ea13646e702633a"
Nov 02 18:09:54.800230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:54.800193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:55.842735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:55.842706    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:09:59.060215 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:09:59.060179    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:09:59.805566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:09:59.805533    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:02.777688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:02.777583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:03.914125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:03.914091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:04.557649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:04.548595    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:04.849736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:04.849653    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:05.132186 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:05.131780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:08.823859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:08.823821    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:09.019864 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:09.019828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:09.656750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:09.656681    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:10.278774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:10.278722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:12.350957 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:12.350872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:13.737935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.737900    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"]
Nov 02 18:10:13.748803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.748775    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.797655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.797540    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"]
Nov 02 18:10:13.805544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.805514    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-dd92ac\"/\"auth-test-model-dd92ac-kserve-self-signed-certs\""
Nov 02 18:10:13.805834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.805807    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-dd92ac\"/\"kube-root-ca.crt\""
Nov 02 18:10:13.806029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.806002    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-dd92ac\"/\"default-dockercfg-8g4ck\""
Nov 02 18:10:13.806114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.805887    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-dd92ac\"/\"openshift-service-ca.crt\""
Nov 02 18:10:13.885428 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.885376    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.885727 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.885706    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.885858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.885839    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cnr\" (UniqueName: \"kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.886111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.886091    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.886238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.886223    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.886346 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.886333    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.911236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.911187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.987663    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.987703    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.987724    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cnr\" (UniqueName: \"kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.987810    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.987830    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.987852    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.988695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.988575    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.989212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.988845    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.989212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.989029    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.991663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.990917    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:13.994215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:13.994190    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:14.052664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:14.052004    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cnr\" (UniqueName: \"kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr\") pod \"auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") " pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:14.089562 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:14.088708    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:14.509899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:14.509869    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"]
Nov 02 18:10:15.095990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:15.095947    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" event={"ID":"07fc693e-b0b5-4be4-9fad-d207ab00f3c4","Type":"ContainerStarted","Data":"35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3"}
Nov 02 18:10:15.096386 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:15.095998    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" event={"ID":"07fc693e-b0b5-4be4-9fad-d207ab00f3c4","Type":"ContainerStarted","Data":"9704052e96fade9eb9acd495106272271822eb8873072aa4830ca02acce87422"}
Nov 02 18:10:15.216191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:15.216145    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" podStartSLOduration=2.2161309 podStartE2EDuration="2.2161309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:15.211601486 +0000 UTC m=+2846.052207432" watchObservedRunningTime="2026-11-02 18:10:15.2161309 +0000 UTC m=+2846.056736879"
Nov 02 18:10:15.476385 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:15.476342    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:16.090022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:16.089992    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:20.166838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:20.166796    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:21.152090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:21.152052    4476 generic.go:356] "Generic (PLEG): container finished" podID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerID="39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67" exitCode=0
Nov 02 18:10:21.152316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:21.152104    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" event={"ID":"f5269bdd-c304-425e-ae66-f6d1b2242bb3","Type":"ContainerDied","Data":"39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"}
Nov 02 18:10:21.152529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:21.152515    4476 scope.go:122] "RemoveContainer" containerID="39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"
Nov 02 18:10:21.154197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:21.154163    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:10:21.154305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:21.154232    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:10:21.154305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:21.154264    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:10:22.176592 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:22.176539    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" event={"ID":"f5269bdd-c304-425e-ae66-f6d1b2242bb3","Type":"ContainerStarted","Data":"542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222"}
Nov 02 18:10:22.317296 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:22.317264    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:10:22.318495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:22.318470    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 02 18:10:22.318612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:22.318535    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 02 18:10:23.317748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:23.317708    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 02 18:10:23.318161 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:23.317768    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 02 18:10:24.090177 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.090103    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:24.102932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.102904    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:24.165654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.164746    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:10:24.204969 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.204939    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:10:24.317417 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.317379    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 02 18:10:24.317657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.317450    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 02 18:10:24.843270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:24.843239    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:25.073776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:25.072702    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"]
Nov 02 18:10:25.076697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:25.076656    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b6c6a0c-fdtrp"]
Nov 02 18:10:25.317231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:25.317187    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 02 18:10:25.317411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:25.317256    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 02 18:10:25.387771 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:25.387732    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99151887-76db-4d9d-a040-5a4e9c275caa" path="/var/lib/kubelet/pods/99151887-76db-4d9d-a040-5a4e9c275caa/volumes"
Nov 02 18:10:26.318005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:26.317970    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 02 18:10:26.318437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:26.318023    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 02 18:10:27.370695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:27.370665    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:10:27.372810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:27.372773    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:10:29.062514 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:10:29.062474    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:10:29.534755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:29.534672    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:34.445220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:34.445158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:38.161205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:38.161112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:39.559448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:39.559403    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:40.046979 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:40.046950    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"]
Nov 02 18:10:40.096090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:40.096054    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5079665e-k9mlg"]
Nov 02 18:10:41.384614 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:41.384577    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8340cef-b8e3-435e-9bfd-e2b87e67420d" path="/var/lib/kubelet/pods/c8340cef-b8e3-435e-9bfd-e2b87e67420d/volumes"
Nov 02 18:10:44.192582 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:44.192551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:48.997579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:48.997542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:50.509022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:50.508988    4476 scope.go:122] "RemoveContainer" containerID="d7c1d9df575c3d15f59b415933a314d931efa0324d505c5c22ebf7301f613c76"
Nov 02 18:10:50.520485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:50.520464    4476 scope.go:122] "RemoveContainer" containerID="6e5736406b52e31944e503df6efdc4c45d1219a146f5bc44ad90231d8d1bc2e2"
Nov 02 18:10:50.530479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:50.530457    4476 scope.go:122] "RemoveContainer" containerID="6b0c55788d4f0346fcf74f8891eb556d2bff1abb2c23326f36a256046cd1d876"
Nov 02 18:10:53.797671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:53.797580    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:54.281272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:54.281238    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:56.167341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:56.167306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:58.652746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:10:58.652700    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:10:59.060915 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:10:59.060827    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:11:00.552693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:00.552623    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:03.807717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:03.807683    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:08.669770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:08.669729    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:13.416685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:13.416652    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:17.103761 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:17.103716    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:18.210096 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:18.209964    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:22.785281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:22.785248    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:22.983050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:22.983003    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:23.198156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:23.198121    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:26.772558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:26.772521    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:27.737315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:27.737280    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:29.062857 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:11:29.062822    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:11:32.482227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:32.482042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:36.478235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:36.478203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:37.177123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:37.177090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:40.893986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:40.893942    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"]
Nov 02 18:11:40.894436 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:40.894188    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy" containerID="cri-o://448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476" gracePeriod=30
Nov 02 18:11:40.944914 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:40.944879    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"]
Nov 02 18:11:40.945163 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:40.945117    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy" containerID="cri-o://80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116" gracePeriod=30
Nov 02 18:11:42.002488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.001196    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:42.899203 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.899165    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"]
Nov 02 18:11:42.903418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.903401    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:42.961145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.961109    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"]
Nov 02 18:11:42.976294 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.976257    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:42.976687 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.976663    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:42.977230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.976711    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:42.977230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.976793    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:42.977230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.976817    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj46n\" (UniqueName: \"kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:42.977230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:42.976857    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.004370 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.004340    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"openshift-service-ca.crt\""
Nov 02 18:11:43.004857 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.004556    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"kube-root-ca.crt\""
Nov 02 18:11:43.005737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.005719    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"default-dockercfg-vnmtz\""
Nov 02 18:11:43.005875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.005860    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"auth-test-model-580a21-kserve-self-signed-certs\""
Nov 02 18:11:43.035498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.035448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:43.078823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.078777    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.078874    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.078908    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.079014    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.079040    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj46n\" (UniqueName: \"kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.079079    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.079242    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.079536    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.079711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.079604    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.082101 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.082077    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.085578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.085529    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.136189 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.136151    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj46n\" (UniqueName: \"kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n\") pod \"auth-test-model-580a21-kserve-9c95f57d5-kr9rv\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") " pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.216540 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.216452    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:43.616572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.616548    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"]
Nov 02 18:11:43.617545 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:11:43.617522    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9450795a_a2b5_4887_bc85_6315b7aba5d0.slice/crio-1ffb726bd76d1aaddb06ddd37696100495aaa5e90fcc49885bb2351e1d372fbf WatchSource:0}: Error finding container 1ffb726bd76d1aaddb06ddd37696100495aaa5e90fcc49885bb2351e1d372fbf: Status 404 returned error can't find the container with id 1ffb726bd76d1aaddb06ddd37696100495aaa5e90fcc49885bb2351e1d372fbf
Nov 02 18:11:43.789020 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.788986    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" event={"ID":"9450795a-a2b5-4887-bc85-6315b7aba5d0","Type":"ContainerStarted","Data":"aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa"}
Nov 02 18:11:43.789195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.789027    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" event={"ID":"9450795a-a2b5-4887-bc85-6315b7aba5d0","Type":"ContainerStarted","Data":"1ffb726bd76d1aaddb06ddd37696100495aaa5e90fcc49885bb2351e1d372fbf"}
Nov 02 18:11:43.903225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:43.903109    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" podStartSLOduration=1.903090781 podStartE2EDuration="1.903090781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:43.901506014 +0000 UTC m=+2934.742111948" watchObservedRunningTime="2026-11-02 18:11:43.903090781 +0000 UTC m=+2934.743696714"
Nov 02 18:11:44.217517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:44.217436    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:46.242157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.242129    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:11:46.298886 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.298854    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:11:46.333483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333454    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333562    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333581    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333602    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333679    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333708    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333742    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.333935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333761    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333791    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333821    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs\") pod \"af345ff6-884c-4b2e-8560-3866eb898f54\" (UID: \"af345ff6-884c-4b2e-8560-3866eb898f54\") "
Nov 02 18:11:46.333935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.333889    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.334424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334336    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.334424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334352    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.334752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334498    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.334752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334520    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.334752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334535    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.334752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334550    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.334752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.334573    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:46.336141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.336120    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:46.336343 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.336323    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.336770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.336747    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:46.336841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.336762    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf" pod "af345ff6-884c-4b2e-8560-3866eb898f54" (UID: "af345ff6-884c-4b2e-8560-3866eb898f54"). InnerVolumeSpecName "kube-api-access-b82lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:46.435510 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435468    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435528    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435603    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435736    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435772    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435864    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435898    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.435990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.435939    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:46.436156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436028    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.436156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436063    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo\") pod \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\" (UID: \"71345efa-c26a-4e14-a8ad-fb745aaa9eef\") "
Nov 02 18:11:46.436332 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436299    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.436403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436329    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.436548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436523    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.436742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436692    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.436843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436759    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.436905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436821    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.436961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436927    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.436961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436944    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af345ff6-884c-4b2e-8560-3866eb898f54-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.436961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436958    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af345ff6-884c-4b2e-8560-3866eb898f54-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.437112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436970    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.437112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.436984    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82lf\" (UniqueName: \"kubernetes.io/projected/af345ff6-884c-4b2e-8560-3866eb898f54-kube-api-access-b82lf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.437112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.437000    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af345ff6-884c-4b2e-8560-3866eb898f54-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.437112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.437018    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.439002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.438679    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "kube-api-access-ggxgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:46.439002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.438919    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:46.439002 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.438944    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:46.439208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.439120    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy" pod "71345efa-c26a-4e14-a8ad-fb745aaa9eef" (UID: "71345efa-c26a-4e14-a8ad-fb745aaa9eef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:46.538563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.538519    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.538563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.538559    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.538818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.538574    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxgw\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-kube-api-access-ggxgw\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.538818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.538590    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.538818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.538604    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71345efa-c26a-4e14-a8ad-fb745aaa9eef-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:11:46.761884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.761766    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:46.817728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.817688    4476 generic.go:356] "Generic (PLEG): container finished" podID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerID="80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116" exitCode=0
Nov 02 18:11:46.817946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.817749    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"
Nov 02 18:11:46.817946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.817743    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" event={"ID":"71345efa-c26a-4e14-a8ad-fb745aaa9eef","Type":"ContainerDied","Data":"80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116"}
Nov 02 18:11:46.817946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.817889    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942" event={"ID":"71345efa-c26a-4e14-a8ad-fb745aaa9eef","Type":"ContainerDied","Data":"34276c35ff371a4b0f30b85fbb995491a25bcc1f26f404f7881b6c92821fd617"}
Nov 02 18:11:46.817946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.817923    4476 scope.go:122] "RemoveContainer" containerID="80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116"
Nov 02 18:11:46.820020 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.819992    4476 generic.go:356] "Generic (PLEG): container finished" podID="af345ff6-884c-4b2e-8560-3866eb898f54" containerID="448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476" exitCode=0
Nov 02 18:11:46.820154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.820033    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"
Nov 02 18:11:46.820154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.820065    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" event={"ID":"af345ff6-884c-4b2e-8560-3866eb898f54","Type":"ContainerDied","Data":"448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476"}
Nov 02 18:11:46.820154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.820119    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p" event={"ID":"af345ff6-884c-4b2e-8560-3866eb898f54","Type":"ContainerDied","Data":"f78e57395ea45da414427efb408196facfe3375d1cc581c5cfee6d5783f558d4"}
Nov 02 18:11:46.831653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.831602    4476 scope.go:122] "RemoveContainer" containerID="80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116"
Nov 02 18:11:46.831989 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:11:46.831970    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116\": container with ID starting with 80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116 not found: ID does not exist" containerID="80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116"
Nov 02 18:11:46.832053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.832000    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116"} err="failed to get container status \"80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116\": rpc error: code = NotFound desc = could not find container \"80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116\": container with ID starting with 80aefc7a7ca1153d69ba33132b62f9a9d935a43e81b6106c4996ba72c6678116 not found: ID does not exist"
Nov 02 18:11:46.832053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.832020    4476 scope.go:122] "RemoveContainer" containerID="448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476"
Nov 02 18:11:46.841934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.841910    4476 scope.go:122] "RemoveContainer" containerID="448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476"
Nov 02 18:11:46.842277 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:11:46.842252    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476\": container with ID starting with 448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476 not found: ID does not exist" containerID="448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476"
Nov 02 18:11:46.842367 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.842295    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476"} err="failed to get container status \"448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476\": rpc error: code = NotFound desc = could not find container \"448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476\": container with ID starting with 448367cb8335ba16007a928a1db57ce762cc3ff78cc1acb49a0b4715efd77476 not found: ID does not exist"
Nov 02 18:11:46.983702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.983669    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"]
Nov 02 18:11:46.989108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:46.989080    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tl942"]
Nov 02 18:11:47.101192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.101099    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"]
Nov 02 18:11:47.106324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.106297    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-w2h8p"]
Nov 02 18:11:47.388376 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.388337    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" path="/var/lib/kubelet/pods/71345efa-c26a-4e14-a8ad-fb745aaa9eef/volumes"
Nov 02 18:11:47.389070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.389050    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" path="/var/lib/kubelet/pods/af345ff6-884c-4b2e-8560-3866eb898f54/volumes"
Nov 02 18:11:47.851408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.851315    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"]
Nov 02 18:11:47.851967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.851948    4476 cpu_manager.go:414] "containerMap: removing container" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy"
Nov 02 18:11:47.851967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.851965    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy"
Nov 02 18:11:47.852157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.851995    4476 cpu_manager.go:414] "containerMap: removing container" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy"
Nov 02 18:11:47.852157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.852001    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy"
Nov 02 18:11:47.852157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.852094    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="71345efa-c26a-4e14-a8ad-fb745aaa9eef" containerName="istio-proxy"
Nov 02 18:11:47.852157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.852106    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="af345ff6-884c-4b2e-8560-3866eb898f54" containerName="istio-proxy"
Nov 02 18:11:47.857004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.856985    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.909760 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.909717    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"kube-root-ca.crt\""
Nov 02 18:11:47.912670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.912622    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"]
Nov 02 18:11:47.955950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.955909    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"default-dockercfg-jzt6h\""
Nov 02 18:11:47.955950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.955907    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"auth-test-model-f7aa2e-kserve-self-signed-certs\""
Nov 02 18:11:47.956219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.955913    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f7aa2e\"/\"openshift-service-ca.crt\""
Nov 02 18:11:47.956542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.956516    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.956616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.956567    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.956766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.956743    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.956885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.956774    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.956939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.956902    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4m46\" (UniqueName: \"kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.957003 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.956963    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:47.999023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:47.998983    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:48.058808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.058766    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.058808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.058813    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.059118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.058898    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4m46\" (UniqueName: \"kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.059118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.058952    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.059118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.059032    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.059118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.059066    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.059575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.059547    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.059853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.059833    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.060497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.060463    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.068716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.063256    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.068716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.063621    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.116490 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.116463    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4m46\" (UniqueName: \"kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46\") pod \"auth-test-model-f7aa2e-kserve-5d6b49585-jvptg\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") " pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.171657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.171550    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:48.570798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.570768    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"]
Nov 02 18:11:48.572911 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:11:48.572880    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13b0849b_2c6e_4601_8f5d_1d715d5f1dbc.slice/crio-c64f768fe7ba4e3cb6f40b52fcc77a22203b136d2d871ad6ca26dbc7fc567879 WatchSource:0}: Error finding container c64f768fe7ba4e3cb6f40b52fcc77a22203b136d2d871ad6ca26dbc7fc567879: Status 404 returned error can't find the container with id c64f768fe7ba4e3cb6f40b52fcc77a22203b136d2d871ad6ca26dbc7fc567879
Nov 02 18:11:48.846192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.846097    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg" event={"ID":"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc","Type":"ContainerStarted","Data":"86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708"}
Nov 02 18:11:48.846192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.846149    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg" event={"ID":"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc","Type":"ContainerStarted","Data":"c64f768fe7ba4e3cb6f40b52fcc77a22203b136d2d871ad6ca26dbc7fc567879"}
Nov 02 18:11:48.960752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:48.960695    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg" podStartSLOduration=1.960680575 podStartE2EDuration="1.960680575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:48.95836613 +0000 UTC m=+2939.798972064" watchObservedRunningTime="2026-11-02 18:11:48.960680575 +0000 UTC m=+2939.801286503"
Nov 02 18:11:49.172572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:49.172529    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:50.575882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:50.575849    4476 scope.go:122] "RemoveContainer" containerID="5b097ed691d5ea8f33d97f48c883ad30906b21069e97820dc2cd2b6b1c67a62b"
Nov 02 18:11:50.588359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:50.588337    4476 scope.go:122] "RemoveContainer" containerID="85e3f6eb15d5f27bf694a8259e9d3dfe5fdb45845dd9b7ec5674ebe29faff638"
Nov 02 18:11:51.511446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:51.511410    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:53.216819 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:53.216784    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:53.229815 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:53.229784    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:53.892691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:53.892660    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:11:56.203257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:56.203219    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:56.493319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:56.493243    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:58.172038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:58.171999    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:58.185331 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:58.185294    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:58.928001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:11:58.927970    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:11:59.060663 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:11:59.060606    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:12:00.919559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:00.919515    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.684941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:05.684907    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.967903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:05.967818    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"]
Nov 02 18:12:05.968107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:05.968077    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy" containerID="cri-o://1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1" gracePeriod=30
Nov 02 18:12:06.029502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:06.029456    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"]
Nov 02 18:12:06.029853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:06.029787    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy" containerID="cri-o://6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30" gracePeriod=30
Nov 02 18:12:09.302986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:09.302942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:10.587315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:10.587263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:11.307213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.307177    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:12:11.361967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.361938    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:12:11.452535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452503    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.452535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452539    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.452875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452586    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.452875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452677    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.452875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452758    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.452875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452805    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.452875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452853    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452899    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452925    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452925    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452937    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.452985    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453014    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453045    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453071    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453101    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.453155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453127    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453175    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453226    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs\") pod \"2232ebee-9107-4f3d-b762-962d009dc173\" (UID: \"2232ebee-9107-4f3d-b762-962d009dc173\") "
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453259    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo\") pod \"269da4e5-61d1-4305-89b5-c52443fc6815\" (UID: \"269da4e5-61d1-4305-89b5-c52443fc6815\") "
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453262    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453377    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453586    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.453747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453699    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.453907    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454080    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454226    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269da4e5-61d1-4305-89b5-c52443fc6815-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454248    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454262    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454275    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454288    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454301    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454313    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2232ebee-9107-4f3d-b762-962d009dc173-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454339    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454354    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.454448    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.455842    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:11.456223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.455955    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:11.457019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.456298    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.457019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.456597    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "kube-api-access-w6skg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:11.457306 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.457282    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:11.458695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.458673    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "kube-api-access-mqt9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:11.458746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.458707    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo" pod "2232ebee-9107-4f3d-b762-962d009dc173" (UID: "2232ebee-9107-4f3d-b762-962d009dc173"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:11.458863 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.458847    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token" pod "269da4e5-61d1-4305-89b5-c52443fc6815" (UID: "269da4e5-61d1-4305-89b5-c52443fc6815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.554970    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.554996    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555006    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269da4e5-61d1-4305-89b5-c52443fc6815-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555016    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2232ebee-9107-4f3d-b762-962d009dc173-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555026    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6skg\" (UniqueName: \"kubernetes.io/projected/269da4e5-61d1-4305-89b5-c52443fc6815-kube-api-access-w6skg\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555037    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555045    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqt9r\" (UniqueName: \"kubernetes.io/projected/2232ebee-9107-4f3d-b762-962d009dc173-kube-api-access-mqt9r\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555054    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2232ebee-9107-4f3d-b762-962d009dc173-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:11.555438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:11.555062    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269da4e5-61d1-4305-89b5-c52443fc6815-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:12:12.019149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.019106    4476 generic.go:356] "Generic (PLEG): container finished" podID="269da4e5-61d1-4305-89b5-c52443fc6815" containerID="1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1" exitCode=0
Nov 02 18:12:12.019596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.019169    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"
Nov 02 18:12:12.019596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.019174    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" event={"ID":"269da4e5-61d1-4305-89b5-c52443fc6815","Type":"ContainerDied","Data":"1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1"}
Nov 02 18:12:12.019596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.019229    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw" event={"ID":"269da4e5-61d1-4305-89b5-c52443fc6815","Type":"ContainerDied","Data":"acc80425b6cd212d69f87e2edc37cf495412ca4a6facb4f7320477fc17fed3bb"}
Nov 02 18:12:12.019596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.019258    4476 scope.go:122] "RemoveContainer" containerID="1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1"
Nov 02 18:12:12.020886 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.020865    4476 generic.go:356] "Generic (PLEG): container finished" podID="2232ebee-9107-4f3d-b762-962d009dc173" containerID="6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30" exitCode=0
Nov 02 18:12:12.021000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.020897    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" event={"ID":"2232ebee-9107-4f3d-b762-962d009dc173","Type":"ContainerDied","Data":"6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30"}
Nov 02 18:12:12.021000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.020922    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"
Nov 02 18:12:12.021000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.020958    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt" event={"ID":"2232ebee-9107-4f3d-b762-962d009dc173","Type":"ContainerDied","Data":"2cacd41a01295ce109956432a7c7bb23cf755e77c8d30ac1f5a3ee97515c08f2"}
Nov 02 18:12:12.033297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.033277    4476 scope.go:122] "RemoveContainer" containerID="1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1"
Nov 02 18:12:12.033901 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:12:12.033875    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1\": container with ID starting with 1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1 not found: ID does not exist" containerID="1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1"
Nov 02 18:12:12.034001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.033912    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1"} err="failed to get container status \"1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1\": rpc error: code = NotFound desc = could not find container \"1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1\": container with ID starting with 1bb70770da8a7972435082453089ce90753f73f84900de7af2199213fcc6c1c1 not found: ID does not exist"
Nov 02 18:12:12.034001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.033937    4476 scope.go:122] "RemoveContainer" containerID="6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30"
Nov 02 18:12:12.044304 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.044284    4476 scope.go:122] "RemoveContainer" containerID="6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30"
Nov 02 18:12:12.044585 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:12:12.044565    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30\": container with ID starting with 6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30 not found: ID does not exist" containerID="6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30"
Nov 02 18:12:12.044691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.044597    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30"} err="failed to get container status \"6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30\": rpc error: code = NotFound desc = could not find container \"6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30\": container with ID starting with 6ca6ececac1bb6bf64a15e060c8ab1527b69a14976c223c5a24fab2f4e5b1e30 not found: ID does not exist"
Nov 02 18:12:12.184294 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.184255    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"]
Nov 02 18:12:12.190856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.190826    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-7k5lt"]
Nov 02 18:12:12.343039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.342955    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"]
Nov 02 18:12:12.346431 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:12.346402    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jvftw"]
Nov 02 18:12:13.383714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:13.383671    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2232ebee-9107-4f3d-b762-962d009dc173" path="/var/lib/kubelet/pods/2232ebee-9107-4f3d-b762-962d009dc173/volumes"
Nov 02 18:12:13.384333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:13.384311    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" path="/var/lib/kubelet/pods/269da4e5-61d1-4305-89b5-c52443fc6815/volumes"
Nov 02 18:12:15.303954 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:15.303867    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:19.999023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:19.998986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:24.764133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:24.764077    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:28.810079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810041    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"]
Nov 02 18:12:28.810657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810640    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy"
Nov 02 18:12:28.810732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810657    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy"
Nov 02 18:12:28.810732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810673    4476 cpu_manager.go:414] "containerMap: removing container" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy"
Nov 02 18:12:28.810732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810678    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy"
Nov 02 18:12:28.810882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810778    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2232ebee-9107-4f3d-b762-962d009dc173" containerName="istio-proxy"
Nov 02 18:12:28.810882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.810786    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="269da4e5-61d1-4305-89b5-c52443fc6815" containerName="istio-proxy"
Nov 02 18:12:28.814018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.813992    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.860964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.860880    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptf2\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.860964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.860929    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.860964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.860953    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.861289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.860987    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.861289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.861025    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.861289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.861041    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.861289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.861075    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.861289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.861102    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.861289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.861157    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.869961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.869895    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-580a21-openshift-default-dockercfg-zr6rg\""
Nov 02 18:12:28.921728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.921687    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"]
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962367    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962457    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962482    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962540    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962588    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962697    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962774    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptf2\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962826    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.962854    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.963716    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.964462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.964396    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.965260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.964697    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.965260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.965241    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.965511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.965490    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.968913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.968883    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:28.969763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:28.969740    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:29.021964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.021930    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:29.022784 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.022759    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptf2\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:29.066763 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:12:29.066620    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:12:29.130962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.130918    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:29.446329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.446279    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:29.607790 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:12:29.607754    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6b56a5_9cc4_461a_98fa_116c89208ffe.slice/crio-7eb98737633414f9085d4bdfd28af579e16c9b9d2903d6f70094ba367b8fdbc3 WatchSource:0}: Error finding container 7eb98737633414f9085d4bdfd28af579e16c9b9d2903d6f70094ba367b8fdbc3: Status 404 returned error can't find the container with id 7eb98737633414f9085d4bdfd28af579e16c9b9d2903d6f70094ba367b8fdbc3
Nov 02 18:12:29.611138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.611100    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:12:29.611234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.611185    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:12:29.611234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.611223    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:12:29.646469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:29.646434    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"]
Nov 02 18:12:30.202931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:30.190173    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" event={"ID":"2d6b56a5-9cc4-461a-98fa-116c89208ffe","Type":"ContainerStarted","Data":"9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9"}
Nov 02 18:12:30.202931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:30.190308    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" event={"ID":"2d6b56a5-9cc4-461a-98fa-116c89208ffe","Type":"ContainerStarted","Data":"7eb98737633414f9085d4bdfd28af579e16c9b9d2903d6f70094ba367b8fdbc3"}
Nov 02 18:12:30.323283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:30.323228    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podStartSLOduration=2.323211615 podStartE2EDuration="2.323211615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:30.321838847 +0000 UTC m=+2981.162444802" watchObservedRunningTime="2026-11-02 18:12:30.323211615 +0000 UTC m=+2981.163817550"
Nov 02 18:12:31.133303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:31.133268    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:31.135070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:31.135043    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:12:31.135221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:31.135097    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 18:12:32.131938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:32.131896    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:12:32.132343 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:32.131967    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 18:12:33.132053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:33.132019    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:12:33.132461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:33.132080    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 18:12:34.131439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:34.131400    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:12:34.131677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:34.131477    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 18:12:34.134808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:34.134773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:35.131994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:35.131952    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 18:12:35.132200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:35.132037    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 18:12:36.258842 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:36.258807    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:36.260844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:36.260815    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:36.261735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:36.261712    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:12:38.828155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:38.828120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:40.378565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:40.378516    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:43.778359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:43.778319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:44.754644 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:44.754588    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:47.336270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:47.336234    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:48.775618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:48.775550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:49.859709 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:49.859681    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:12:49.870019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:49.869983    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:12:50.065354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:50.065294    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:53.460413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:53.460378    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:55.575868 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:55.575833    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:55.809741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:55.809703    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:56.241794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:56.241759    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:56.827820 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:56.827782    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:58.390019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:12:58.389987    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:59.062371 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:12:59.062332    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:13:02.184952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:02.184865    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:03.126084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:03.126048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:07.765794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:07.765741    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:12.500271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:12.500239    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:17.167734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:17.167682    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:21.399287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:21.399250    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:21.961583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:21.961531    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:26.641856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:26.641812    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:29.062149 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:13:29.062111    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:13:31.347714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:31.347679    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:35.975247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:35.975211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:39.960093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:39.960053    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"]
Nov 02 18:13:39.960665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:39.960336    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" podUID="eb16be8a-51ff-42c5-81fe-03cd76da5890" containerName="maas-api" containerID="cri-o://e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997" gracePeriod=30
Nov 02 18:13:40.113931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.113894    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:40.259341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.259299    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:13:40.300509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.300472    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\") pod \"eb16be8a-51ff-42c5-81fe-03cd76da5890\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") "
Nov 02 18:13:40.300719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.300556    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl\" (UniqueName: \"kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl\") pod \"eb16be8a-51ff-42c5-81fe-03cd76da5890\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") "
Nov 02 18:13:40.300719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.300681    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs\") pod \"eb16be8a-51ff-42c5-81fe-03cd76da5890\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") "
Nov 02 18:13:40.300853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.300735    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca\") pod \"eb16be8a-51ff-42c5-81fe-03cd76da5890\" (UID: \"eb16be8a-51ff-42c5-81fe-03cd76da5890\") "
Nov 02 18:13:40.301381 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.301325    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca" pod "eb16be8a-51ff-42c5-81fe-03cd76da5890" (UID: "eb16be8a-51ff-42c5-81fe-03cd76da5890"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:40.303675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.303647    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs" pod "eb16be8a-51ff-42c5-81fe-03cd76da5890" (UID: "eb16be8a-51ff-42c5-81fe-03cd76da5890"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:40.303786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.303667    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl" pod "eb16be8a-51ff-42c5-81fe-03cd76da5890" (UID: "eb16be8a-51ff-42c5-81fe-03cd76da5890"). InnerVolumeSpecName "kube-api-access-rkjrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:40.303847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.303791    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls" pod "eb16be8a-51ff-42c5-81fe-03cd76da5890" (UID: "eb16be8a-51ff-42c5-81fe-03cd76da5890"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:40.408395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.408356    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:40.408395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.408394    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjrl\" (UniqueName: \"kubernetes.io/projected/eb16be8a-51ff-42c5-81fe-03cd76da5890-kube-api-access-rkjrl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:40.408613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.408410    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb16be8a-51ff-42c5-81fe-03cd76da5890-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:40.408613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.408422    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb16be8a-51ff-42c5-81fe-03cd76da5890-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:13:40.734283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.734236    4476 generic.go:356] "Generic (PLEG): container finished" podID="eb16be8a-51ff-42c5-81fe-03cd76da5890" containerID="e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997" exitCode=0
Nov 02 18:13:40.734483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.734382    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" event={"ID":"eb16be8a-51ff-42c5-81fe-03cd76da5890","Type":"ContainerDied","Data":"e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997"}
Nov 02 18:13:40.734483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.734448    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz" event={"ID":"eb16be8a-51ff-42c5-81fe-03cd76da5890","Type":"ContainerDied","Data":"bca543e49e879af6c5d58a9d4204854c1903028741595fc3c6b46408a3a996b1"}
Nov 02 18:13:40.734483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.734478    4476 scope.go:122] "RemoveContainer" containerID="e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997"
Nov 02 18:13:40.734684 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.734376    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"
Nov 02 18:13:40.748856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.748654    4476 scope.go:122] "RemoveContainer" containerID="e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997"
Nov 02 18:13:40.749234 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:13:40.749201    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997\": container with ID starting with e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997 not found: ID does not exist" containerID="e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997"
Nov 02 18:13:40.749356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.749245    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997"} err="failed to get container status \"e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997\": rpc error: code = NotFound desc = could not find container \"e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997\": container with ID starting with e1dee9190c933cbd3cc0d6091f39a71c5f0415e7641e6a90f8a93ac6f724b997 not found: ID does not exist"
Nov 02 18:13:40.899900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.899862    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"]
Nov 02 18:13:40.903533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:40.903505    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb1633-5ffc5f779-6tlbz"]
Nov 02 18:13:41.384228 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:41.384192    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb16be8a-51ff-42c5-81fe-03cd76da5890" path="/var/lib/kubelet/pods/eb16be8a-51ff-42c5-81fe-03cd76da5890/volumes"
Nov 02 18:13:41.862119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:41.862076    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:42.312567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:42.312482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:42.488902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:42.488866    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:43.908488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:43.908453    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:44.545372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:44.545335    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:45.282807 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:45.282775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:46.886446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:46.886406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:50.913250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:50.913196    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:51.180933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:51.180896    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:51.720741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:51.720703    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:51.902330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:51.902285    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.577762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.577727    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"]
Nov 02 18:13:52.578300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.578286    4476 cpu_manager.go:414] "containerMap: removing container" podUID="eb16be8a-51ff-42c5-81fe-03cd76da5890" containerName="maas-api"
Nov 02 18:13:52.578341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.578302    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb16be8a-51ff-42c5-81fe-03cd76da5890" containerName="maas-api"
Nov 02 18:13:52.578406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.578397    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb16be8a-51ff-42c5-81fe-03cd76da5890" containerName="maas-api"
Nov 02 18:13:52.582844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.582818    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.586591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.586564    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.586721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.586657    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.586768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.586725    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmpgl\" (UniqueName: \"kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.586912 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.586870    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.586970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.586943    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.587027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.586991    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.639974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.639941    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"]
Nov 02 18:13:52.683828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.683792    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"default-dockercfg-vjh7q\""
Nov 02 18:13:52.684034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.683849    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"openshift-service-ca.crt\""
Nov 02 18:13:52.684034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.683873    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"auth-test-model-c82472-kserve-self-signed-certs\""
Nov 02 18:13:52.684934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.684921    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"kube-root-ca.crt\""
Nov 02 18:13:52.687934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.687909    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.687992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.687960    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688000    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688068 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688048    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688091    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688113    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmpgl\" (UniqueName: \"kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688358    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688415    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.688525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.688509    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.690490 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.690471    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.690680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.690664    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.750238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.750203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.788902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.788864    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmpgl\" (UniqueName: \"kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl\") pod \"auth-test-model-c82472-kserve-5dbb4456c7-7zh5l\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") " pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:52.895451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:52.895420    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:53.191125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:53.191016    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:53.290752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:53.290724    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"]
Nov 02 18:13:53.292205 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:13:53.292178    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a961254_cbfe_4979_b041_12879d6ab4fa.slice/crio-488ba740e2fd19ac561610e46de49ae425bb7618924c3ab0a6f53bf1bb51ea01 WatchSource:0}: Error finding container 488ba740e2fd19ac561610e46de49ae425bb7618924c3ab0a6f53bf1bb51ea01: Status 404 returned error can't find the container with id 488ba740e2fd19ac561610e46de49ae425bb7618924c3ab0a6f53bf1bb51ea01
Nov 02 18:13:53.842669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:53.842601    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l" event={"ID":"2a961254-cbfe-4979-b041-12879d6ab4fa","Type":"ContainerStarted","Data":"637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769"}
Nov 02 18:13:53.843085 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:53.842683    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l" event={"ID":"2a961254-cbfe-4979-b041-12879d6ab4fa","Type":"ContainerStarted","Data":"488ba740e2fd19ac561610e46de49ae425bb7618924c3ab0a6f53bf1bb51ea01"}
Nov 02 18:13:53.895565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:53.895529    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:13:53.957300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:53.957239    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l" podStartSLOduration=1.95722014 podStartE2EDuration="1.95722014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:53.95567268 +0000 UTC m=+3064.796278613" watchObservedRunningTime="2026-11-02 18:13:53.95722014 +0000 UTC m=+3064.797826076"
Nov 02 18:13:54.863172 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:54.863130    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"]
Nov 02 18:13:54.863717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:54.863393    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" containerID="cri-o://96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01" gracePeriod=30
Nov 02 18:13:56.493047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:56.493010    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:57.178366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:57.178301    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:57.999712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:13:57.999669    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:59.063811 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:13:59.063774    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:14:00.024387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.024354    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 18:14:00.024554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.024419    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 18:14:00.135028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.134962    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:14:00.181281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181246    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181451    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181524    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181552    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181578    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181609    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181658    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181689    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data\") pod \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\" (UID: \"c1449759-d1a4-46c9-a8bc-f5a71fda118b\") "
Nov 02 18:14:00.181765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.181743    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:14:00.182136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.182049    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:00.182136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.182107    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.182341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.182264    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:00.182503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.182483    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:00.182668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.182613    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:00.184475 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.184450    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:14:00.184475 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.184460    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:00.184893 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.184873    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "kube-api-access-gnh9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:00.185713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.185687    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo" pod "c1449759-d1a4-46c9-a8bc-f5a71fda118b" (UID: "c1449759-d1a4-46c9-a8bc-f5a71fda118b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:14:00.283762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283728    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.283762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283757    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnh9m\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-kube-api-access-gnh9m\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.283762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283768    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.284095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283777    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.284095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283788    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.284095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283796    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.284095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283806    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.284095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.283815    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1449759-d1a4-46c9-a8bc-f5a71fda118b-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.901085 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.901050    4476 generic.go:356] "Generic (PLEG): container finished" podID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerID="96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01" exitCode=0
Nov 02 18:14:00.901246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.901096    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"
Nov 02 18:14:00.901246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.901135    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" event={"ID":"c1449759-d1a4-46c9-a8bc-f5a71fda118b","Type":"ContainerDied","Data":"96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01"}
Nov 02 18:14:00.901246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.901192    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7" event={"ID":"c1449759-d1a4-46c9-a8bc-f5a71fda118b","Type":"ContainerDied","Data":"4f478445ac75854a40f7daf7b39ad55dfddf14e5dd2a357037f1cf111a9b28bc"}
Nov 02 18:14:00.901246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.901218    4476 scope.go:122] "RemoveContainer" containerID="96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01"
Nov 02 18:14:00.913061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.913041    4476 scope.go:122] "RemoveContainer" containerID="96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01"
Nov 02 18:14:00.913363 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:14:00.913344    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01\": container with ID starting with 96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01 not found: ID does not exist" containerID="96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01"
Nov 02 18:14:00.913416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:00.913373    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01"} err="failed to get container status \"96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01\": rpc error: code = NotFound desc = could not find container \"96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01\": container with ID starting with 96e5a79623d758bcb38cac1d624f61afd3dd27d176d7867806648230eb84eb01 not found: ID does not exist"
Nov 02 18:14:01.065127 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:01.065087    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"]
Nov 02 18:14:01.068448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:01.068422    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-6kgw7"]
Nov 02 18:14:01.385146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:01.385104    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" path="/var/lib/kubelet/pods/c1449759-d1a4-46c9-a8bc-f5a71fda118b/volumes"
Nov 02 18:14:02.007699 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:02.007657    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:02.387322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:02.387280    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:02.854766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:02.850986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:02.895909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:02.895872    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:14:02.909027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:02.909003    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:14:02.934918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:02.934887    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:14:03.565434 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:03.565395    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:04.043829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:04.043762    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:04.933398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:04.933368    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:05.539091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:05.539050    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:05.793199 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:05.793103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:08.001387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:08.001349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:09.812237 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:09.812192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:10.095152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:10.095052    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:12.942200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:12.942161    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:13.144692 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:13.144660    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:13.331873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:13.331753    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:15.466042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:15.465877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:17.643298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:17.643244    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:22.368513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:22.368433    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:27.164091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:27.164032    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:29.062882 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:14:29.062845    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:14:31.909400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:31.909296    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:36.591913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:36.591879    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:38.470197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:38.470163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:38.650011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:38.649971    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:41.289365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:41.289318    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:45.934982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:45.934915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:46.598505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:46.598463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:47.235153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:47.235114    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:47.453019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:47.452968    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:50.702778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:50.702576    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:52.086351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:52.086298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:52.285018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:52.284965    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:52.475417 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:52.475364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:55.300837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:55.300797    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:55.546243 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:55.544968    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:56.003328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:56.003291    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:57.518739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:14:57.518701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:59.062046 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:14:59.062003    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:15:00.173292 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.173253    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"]
Nov 02 18:15:00.174115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.174096    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy"
Nov 02 18:15:00.174167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.174120    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy"
Nov 02 18:15:00.174327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.174316    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1449759-d1a4-46c9-a8bc-f5a71fda118b" containerName="istio-proxy"
Nov 02 18:15:00.179398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.179379    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"]
Nov 02 18:15:00.179523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.179509    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.183693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.183669    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.200885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.200859    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"]
Nov 02 18:15:00.204845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.204829    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.222597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.222573    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"]
Nov 02 18:15:00.226224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.226208    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"]
Nov 02 18:15:00.226336 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.226318    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.229888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.229870    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.256779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.256750    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9tz\" (UniqueName: \"kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.256914 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.256825    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.256914 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.256873    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbd84\" (UniqueName: \"kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.256995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.256958    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.271302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.271270    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"]
Nov 02 18:15:00.274745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.274720    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"]
Nov 02 18:15:00.342053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.342024    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:15:00.357828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.357782    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.358137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358107    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.358265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358206    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.358265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358245    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtz7g\" (UniqueName: \"kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.358389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358321    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzt8\" (UniqueName: \"kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.358448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358391    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9tz\" (UniqueName: \"kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.358448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358421    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.358541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358463    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhwk\" (UniqueName: \"kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.358541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358519    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.358685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358542    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.358685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.358550    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbd84\" (UniqueName: \"kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.359058 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.359037    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"]
Nov 02 18:15:00.359347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.359324    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.372848 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.372821    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"]
Nov 02 18:15:00.379246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.379219    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"]
Nov 02 18:15:00.416290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.416263    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9tz\" (UniqueName: \"kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.416537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.416512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbd84\" (UniqueName: \"kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.460609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.460529    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhwk\" (UniqueName: \"kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.460797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.460670    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.460797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.460689    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.460797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.460714    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtz7g\" (UniqueName: \"kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.460797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.460760    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzt8\" (UniqueName: \"kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.461040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.460802    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.461376 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.461354    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.461420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.461401    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.461456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.461418    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.494473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.494439    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:00.502882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.502858    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:00.521927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.520053    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzt8\" (UniqueName: \"kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.521927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.520433    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhwk\" (UniqueName: \"kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.521927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.521058    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtz7g\" (UniqueName: \"kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.540457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.540396    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:00.548174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.548148    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:00.775259 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.775188    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:00.818486 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.818448    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:00.890973 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.890727    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"]
Nov 02 18:15:00.895055 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:15:00.894847    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e25d042_df0e_4c8b_aec0_6870982714e2.slice/crio-355930703b0f9ed696af60f550814c6b875eefc92e0ea89f593b6355fbdaf29a WatchSource:0}: Error finding container 355930703b0f9ed696af60f550814c6b875eefc92e0ea89f593b6355fbdaf29a: Status 404 returned error can't find the container with id 355930703b0f9ed696af60f550814c6b875eefc92e0ea89f593b6355fbdaf29a
Nov 02 18:15:00.942708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.942677    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"]
Nov 02 18:15:00.944149 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:15:00.944119    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a3bb17_4a0d_4377_b731_33bc3de59a96.slice/crio-db80833fccd8bcd96fe7d9b303f2c0c7dab9730135ce6bf6c9e9d2b55c5c278f WatchSource:0}: Error finding container db80833fccd8bcd96fe7d9b303f2c0c7dab9730135ce6bf6c9e9d2b55c5c278f: Status 404 returned error can't find the container with id db80833fccd8bcd96fe7d9b303f2c0c7dab9730135ce6bf6c9e9d2b55c5c278f
Nov 02 18:15:00.991405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.991288    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"]
Nov 02 18:15:00.995698 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:15:00.993704    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d7331b_5dd1_4283_a58f_f427294e8b30.slice/crio-0f492f5a2c5c2d28ee37f6e928d8df17b0cb83658abab02a521452e8b6ddeb79 WatchSource:0}: Error finding container 0f492f5a2c5c2d28ee37f6e928d8df17b0cb83658abab02a521452e8b6ddeb79: Status 404 returned error can't find the container with id 0f492f5a2c5c2d28ee37f6e928d8df17b0cb83658abab02a521452e8b6ddeb79
Nov 02 18:15:00.996158 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:00.996129    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"]
Nov 02 18:15:00.998207 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:15:00.998022    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33db761_c384_44a6_9765_0abeae673657.slice/crio-3a3a54ed35439a2b1e4b70fa121d1a4f5044bcda9965822492647c0a1dda3cdd WatchSource:0}: Error finding container 3a3a54ed35439a2b1e4b70fa121d1a4f5044bcda9965822492647c0a1dda3cdd: Status 404 returned error can't find the container with id 3a3a54ed35439a2b1e4b70fa121d1a4f5044bcda9965822492647c0a1dda3cdd
Nov 02 18:15:01.209895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.209871    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"]
Nov 02 18:15:01.210343 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:15:01.210319    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17979935_c344_4bad_a54d_3a47ed7b294f.slice/crio-59c8740cf853765430bf2d4b6675c89c27b90b8f4df014ae094a290f3eaca2b3 WatchSource:0}: Error finding container 59c8740cf853765430bf2d4b6675c89c27b90b8f4df014ae094a290f3eaca2b3: Status 404 returned error can't find the container with id 59c8740cf853765430bf2d4b6675c89c27b90b8f4df014ae094a290f3eaca2b3
Nov 02 18:15:01.374499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.374466    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" event={"ID":"1e25d042-df0e-4c8b-aec0-6870982714e2","Type":"ContainerStarted","Data":"355930703b0f9ed696af60f550814c6b875eefc92e0ea89f593b6355fbdaf29a"}
Nov 02 18:15:01.391477 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.391438    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" event={"ID":"17979935-c344-4bad-a54d-3a47ed7b294f","Type":"ContainerStarted","Data":"59c8740cf853765430bf2d4b6675c89c27b90b8f4df014ae094a290f3eaca2b3"}
Nov 02 18:15:01.391688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.391502    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerStarted","Data":"709c2043b0a1a086f1102e502e32108624570904ef5a7a7cbe42db742da16d07"}
Nov 02 18:15:01.391688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.391532    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerStarted","Data":"0f492f5a2c5c2d28ee37f6e928d8df17b0cb83658abab02a521452e8b6ddeb79"}
Nov 02 18:15:01.391688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.391553    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerStarted","Data":"e23c60bcc26661c491f9b2a1734750fdb160c61451039c7935d2c3ec8bb81a25"}
Nov 02 18:15:01.391688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.391575    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerStarted","Data":"db80833fccd8bcd96fe7d9b303f2c0c7dab9730135ce6bf6c9e9d2b55c5c278f"}
Nov 02 18:15:01.391688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.391597    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" event={"ID":"b33db761-c384-44a6-9765-0abeae673657","Type":"ContainerStarted","Data":"3a3a54ed35439a2b1e4b70fa121d1a4f5044bcda9965822492647c0a1dda3cdd"}
Nov 02 18:15:01.493262 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.493208    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" podStartSLOduration=1.493192457 podStartE2EDuration="1.493192457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:01.489781488 +0000 UTC m=+3132.330387421" watchObservedRunningTime="2026-11-02 18:15:01.493192457 +0000 UTC m=+3132.333798391"
Nov 02 18:15:01.609220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:01.609097    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" podStartSLOduration=1.609077294 podStartE2EDuration="1.609077294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:01.601169454 +0000 UTC m=+3132.441775388" watchObservedRunningTime="2026-11-02 18:15:01.609077294 +0000 UTC m=+3132.449683232"
Nov 02 18:15:02.394806 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.394771    4476 generic.go:356] "Generic (PLEG): container finished" podID="b33db761-c384-44a6-9765-0abeae673657" containerID="1b8249c934920a61f87f9ddf36b3cb2f74835b1bff155cdcb84be9976ffe818f" exitCode=60
Nov 02 18:15:02.395231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.394847    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" event={"ID":"b33db761-c384-44a6-9765-0abeae673657","Type":"ContainerDied","Data":"1b8249c934920a61f87f9ddf36b3cb2f74835b1bff155cdcb84be9976ffe818f"}
Nov 02 18:15:02.395231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.395090    4476 scope.go:122] "RemoveContainer" containerID="1b8249c934920a61f87f9ddf36b3cb2f74835b1bff155cdcb84be9976ffe818f"
Nov 02 18:15:02.396839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.396821    4476 generic.go:356] "Generic (PLEG): container finished" podID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerID="59f47ef0ea4891e1379fde8e05ca7b420b3333491fa56e3f7e27121104f5cc56" exitCode=60
Nov 02 18:15:02.396924 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.396904    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" event={"ID":"1e25d042-df0e-4c8b-aec0-6870982714e2","Type":"ContainerDied","Data":"59f47ef0ea4891e1379fde8e05ca7b420b3333491fa56e3f7e27121104f5cc56"}
Nov 02 18:15:02.397128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.397110    4476 scope.go:122] "RemoveContainer" containerID="59f47ef0ea4891e1379fde8e05ca7b420b3333491fa56e3f7e27121104f5cc56"
Nov 02 18:15:02.398695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.398659    4476 generic.go:356] "Generic (PLEG): container finished" podID="17979935-c344-4bad-a54d-3a47ed7b294f" containerID="50248d398dbcd2f2b9f529a7fc77bb60ce2cf0eb3c271acf6eeffbc247ef8335" exitCode=60
Nov 02 18:15:02.398769 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.398745    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" event={"ID":"17979935-c344-4bad-a54d-3a47ed7b294f","Type":"ContainerDied","Data":"50248d398dbcd2f2b9f529a7fc77bb60ce2cf0eb3c271acf6eeffbc247ef8335"}
Nov 02 18:15:02.398954 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.398938    4476 scope.go:122] "RemoveContainer" containerID="50248d398dbcd2f2b9f529a7fc77bb60ce2cf0eb3c271acf6eeffbc247ef8335"
Nov 02 18:15:02.400718 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.400696    4476 generic.go:356] "Generic (PLEG): container finished" podID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerID="709c2043b0a1a086f1102e502e32108624570904ef5a7a7cbe42db742da16d07" exitCode=60
Nov 02 18:15:02.400820 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.400766    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerDied","Data":"709c2043b0a1a086f1102e502e32108624570904ef5a7a7cbe42db742da16d07"}
Nov 02 18:15:02.400972 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.400956    4476 scope.go:122] "RemoveContainer" containerID="709c2043b0a1a086f1102e502e32108624570904ef5a7a7cbe42db742da16d07"
Nov 02 18:15:02.402356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.402336    4476 generic.go:356] "Generic (PLEG): container finished" podID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerID="e23c60bcc26661c491f9b2a1734750fdb160c61451039c7935d2c3ec8bb81a25" exitCode=60
Nov 02 18:15:02.402465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.402373    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerDied","Data":"e23c60bcc26661c491f9b2a1734750fdb160c61451039c7935d2c3ec8bb81a25"}
Nov 02 18:15:02.402597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:02.402582    4476 scope.go:122] "RemoveContainer" containerID="e23c60bcc26661c491f9b2a1734750fdb160c61451039c7935d2c3ec8bb81a25"
Nov 02 18:15:03.411841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.411807    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" event={"ID":"1e25d042-df0e-4c8b-aec0-6870982714e2","Type":"ContainerStarted","Data":"55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e"}
Nov 02 18:15:03.413956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.413931    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" event={"ID":"17979935-c344-4bad-a54d-3a47ed7b294f","Type":"ContainerStarted","Data":"55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f"}
Nov 02 18:15:03.415994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.415964    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerStarted","Data":"6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56"}
Nov 02 18:15:03.418019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.417993    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerStarted","Data":"27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922"}
Nov 02 18:15:03.419988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.419962    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" event={"ID":"b33db761-c384-44a6-9765-0abeae673657","Type":"ContainerStarted","Data":"cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502"}
Nov 02 18:15:03.528534 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.528487    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" podStartSLOduration=3.528472079 podStartE2EDuration="3.528472079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.523365538 +0000 UTC m=+3134.363971495" watchObservedRunningTime="2026-11-02 18:15:03.528472079 +0000 UTC m=+3134.369078013"
Nov 02 18:15:03.641064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.641018    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" podStartSLOduration=3.640999366 podStartE2EDuration="3.640999366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.637777338 +0000 UTC m=+3134.478383276" watchObservedRunningTime="2026-11-02 18:15:03.640999366 +0000 UTC m=+3134.481605303"
Nov 02 18:15:03.753397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:03.753342    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" podStartSLOduration=3.753326704 podStartE2EDuration="3.753326704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.751570805 +0000 UTC m=+3134.592176741" watchObservedRunningTime="2026-11-02 18:15:03.753326704 +0000 UTC m=+3134.593932638"
Nov 02 18:15:04.428738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.428694    4476 generic.go:356] "Generic (PLEG): container finished" podID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerID="55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e" exitCode=60
Nov 02 18:15:04.429246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.428780    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" event={"ID":"1e25d042-df0e-4c8b-aec0-6870982714e2","Type":"ContainerDied","Data":"55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e"}
Nov 02 18:15:04.429246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.428838    4476 scope.go:122] "RemoveContainer" containerID="59f47ef0ea4891e1379fde8e05ca7b420b3333491fa56e3f7e27121104f5cc56"
Nov 02 18:15:04.429246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.429080    4476 scope.go:122] "RemoveContainer" containerID="55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e"
Nov 02 18:15:04.429452 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:04.429427    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s_odh-ai-gateway-infra(1e25d042-df0e-4c8b-aec0-6870982714e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2"
Nov 02 18:15:04.430884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.430859    4476 generic.go:356] "Generic (PLEG): container finished" podID="17979935-c344-4bad-a54d-3a47ed7b294f" containerID="55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f" exitCode=60
Nov 02 18:15:04.431023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.430947    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" event={"ID":"17979935-c344-4bad-a54d-3a47ed7b294f","Type":"ContainerDied","Data":"55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f"}
Nov 02 18:15:04.431164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.431151    4476 scope.go:122] "RemoveContainer" containerID="55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f"
Nov 02 18:15:04.431459 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:04.431439    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm_odh-ai-gateway-infra(17979935-c344-4bad-a54d-3a47ed7b294f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" podUID="17979935-c344-4bad-a54d-3a47ed7b294f"
Nov 02 18:15:04.432785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.432760    4476 generic.go:356] "Generic (PLEG): container finished" podID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerID="6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56" exitCode=60
Nov 02 18:15:04.432883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.432789    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerDied","Data":"6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56"}
Nov 02 18:15:04.433062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.433047    4476 scope.go:122] "RemoveContainer" containerID="6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56"
Nov 02 18:15:04.433321 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:04.433300    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q_odh-ai-gateway-infra(c2d7331b-5dd1-4283-a58f-f427294e8b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30"
Nov 02 18:15:04.434706 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.434686    4476 generic.go:356] "Generic (PLEG): container finished" podID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerID="27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922" exitCode=60
Nov 02 18:15:04.434810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.434723    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerDied","Data":"27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922"}
Nov 02 18:15:04.434919 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.434905    4476 scope.go:122] "RemoveContainer" containerID="27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922"
Nov 02 18:15:04.435129 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:04.435111    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4_odh-ai-gateway-infra(86a3bb17-4a0d-4377-b731-33bc3de59a96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96"
Nov 02 18:15:04.436609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.436588    4476 generic.go:356] "Generic (PLEG): container finished" podID="b33db761-c384-44a6-9765-0abeae673657" containerID="cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502" exitCode=60
Nov 02 18:15:04.436749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.436669    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" event={"ID":"b33db761-c384-44a6-9765-0abeae673657","Type":"ContainerDied","Data":"cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502"}
Nov 02 18:15:04.436943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.436930    4476 scope.go:122] "RemoveContainer" containerID="cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502"
Nov 02 18:15:04.437227 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:04.437198    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh_odh-ai-gateway-infra(b33db761-c384-44a6-9765-0abeae673657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" podUID="b33db761-c384-44a6-9765-0abeae673657"
Nov 02 18:15:04.442839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.442821    4476 scope.go:122] "RemoveContainer" containerID="50248d398dbcd2f2b9f529a7fc77bb60ce2cf0eb3c271acf6eeffbc247ef8335"
Nov 02 18:15:04.456709 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.456680    4476 scope.go:122] "RemoveContainer" containerID="709c2043b0a1a086f1102e502e32108624570904ef5a7a7cbe42db742da16d07"
Nov 02 18:15:04.471962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.471941    4476 scope.go:122] "RemoveContainer" containerID="e23c60bcc26661c491f9b2a1734750fdb160c61451039c7935d2c3ec8bb81a25"
Nov 02 18:15:04.485910 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:04.485884    4476 scope.go:122] "RemoveContainer" containerID="1b8249c934920a61f87f9ddf36b3cb2f74835b1bff155cdcb84be9976ffe818f"
Nov 02 18:15:05.439359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:05.439324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:05.446463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:05.446425    4476 scope.go:122] "RemoveContainer" containerID="55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e"
Nov 02 18:15:05.446874 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:05.446832    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s_odh-ai-gateway-infra(1e25d042-df0e-4c8b-aec0-6870982714e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2"
Nov 02 18:15:05.448734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:05.448712    4476 scope.go:122] "RemoveContainer" containerID="55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f"
Nov 02 18:15:05.449104 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:05.449075    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm_odh-ai-gateway-infra(17979935-c344-4bad-a54d-3a47ed7b294f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" podUID="17979935-c344-4bad-a54d-3a47ed7b294f"
Nov 02 18:15:05.450924 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:05.450891    4476 scope.go:122] "RemoveContainer" containerID="6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56"
Nov 02 18:15:05.451203 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:05.451164    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q_odh-ai-gateway-infra(c2d7331b-5dd1-4283-a58f-f427294e8b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30"
Nov 02 18:15:05.453043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:05.453023    4476 scope.go:122] "RemoveContainer" containerID="27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922"
Nov 02 18:15:05.453313 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:05.453282    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4_odh-ai-gateway-infra(86a3bb17-4a0d-4377-b731-33bc3de59a96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96"
Nov 02 18:15:05.455062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:05.455043    4476 scope.go:122] "RemoveContainer" containerID="cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502"
Nov 02 18:15:05.455309 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:05.455282    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh_odh-ai-gateway-infra(b33db761-c384-44a6-9765-0abeae673657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" podUID="b33db761-c384-44a6-9765-0abeae673657"
Nov 02 18:15:10.161319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:10.161282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:14.745780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:14.745718    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:15.376187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:15.376152    4476 scope.go:122] "RemoveContainer" containerID="27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922"
Nov 02 18:15:16.376502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:16.376476    4476 scope.go:122] "RemoveContainer" containerID="cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502"
Nov 02 18:15:16.543915 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:16.543880    4476 generic.go:356] "Generic (PLEG): container finished" podID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerID="fe676a028338c57d5e7063403678b82e9447283caaef22ee783f646037df071c" exitCode=60
Nov 02 18:15:16.543915 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:16.543902    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerDied","Data":"fe676a028338c57d5e7063403678b82e9447283caaef22ee783f646037df071c"}
Nov 02 18:15:16.544136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:16.543979    4476 scope.go:122] "RemoveContainer" containerID="27f391942348c957006ad4e098c563e101fba43ceeb1d27468d7f7978d8cc922"
Nov 02 18:15:16.544396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:16.544372    4476 scope.go:122] "RemoveContainer" containerID="fe676a028338c57d5e7063403678b82e9447283caaef22ee783f646037df071c"
Nov 02 18:15:16.544763 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:16.544736    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4_odh-ai-gateway-infra(86a3bb17-4a0d-4377-b731-33bc3de59a96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96"
Nov 02 18:15:17.376313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.376275    4476 scope.go:122] "RemoveContainer" containerID="55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e"
Nov 02 18:15:17.376596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.376578    4476 scope.go:122] "RemoveContainer" containerID="55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f"
Nov 02 18:15:17.490561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.490483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:17.559193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.559157    4476 generic.go:356] "Generic (PLEG): container finished" podID="b33db761-c384-44a6-9765-0abeae673657" containerID="44698cc344a314df2dda4b6c985598a5278bf8407ec01de75eee4db27a872d3b" exitCode=60
Nov 02 18:15:17.559398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.559221    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" event={"ID":"b33db761-c384-44a6-9765-0abeae673657","Type":"ContainerDied","Data":"44698cc344a314df2dda4b6c985598a5278bf8407ec01de75eee4db27a872d3b"}
Nov 02 18:15:17.559398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.559296    4476 scope.go:122] "RemoveContainer" containerID="cf064fed3386e1353a5c03437788b6778c2923cb322302aa6fbaf4f167925502"
Nov 02 18:15:17.559754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.559737    4476 scope.go:122] "RemoveContainer" containerID="44698cc344a314df2dda4b6c985598a5278bf8407ec01de75eee4db27a872d3b"
Nov 02 18:15:17.560064 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:17.560041    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh_odh-ai-gateway-infra(b33db761-c384-44a6-9765-0abeae673657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" podUID="b33db761-c384-44a6-9765-0abeae673657"
Nov 02 18:15:17.674004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.673954    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"]
Nov 02 18:15:17.835644 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.835602    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:17.878654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.878606    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz\" (UniqueName: \"kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz\") pod \"86a3bb17-4a0d-4377-b731-33bc3de59a96\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") "
Nov 02 18:15:17.878858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.878796    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca\") pod \"86a3bb17-4a0d-4377-b731-33bc3de59a96\" (UID: \"86a3bb17-4a0d-4377-b731-33bc3de59a96\") "
Nov 02 18:15:17.879183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.879159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca" pod "86a3bb17-4a0d-4377-b731-33bc3de59a96" (UID: "86a3bb17-4a0d-4377-b731-33bc3de59a96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:17.881254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.881229    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz" pod "86a3bb17-4a0d-4377-b731-33bc3de59a96" (UID: "86a3bb17-4a0d-4377-b731-33bc3de59a96"). InnerVolumeSpecName "kube-api-access-5g9tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:17.980244 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.980199    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g9tz\" (UniqueName: \"kubernetes.io/projected/86a3bb17-4a0d-4377-b731-33bc3de59a96-kube-api-access-5g9tz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:17.980244 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:17.980234    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a3bb17-4a0d-4377-b731-33bc3de59a96-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:18.376318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.376294    4476 scope.go:122] "RemoveContainer" containerID="6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56"
Nov 02 18:15:18.568212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.568182    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"
Nov 02 18:15:18.568712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.568213    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4" event={"ID":"86a3bb17-4a0d-4377-b731-33bc3de59a96","Type":"ContainerDied","Data":"db80833fccd8bcd96fe7d9b303f2c0c7dab9730135ce6bf6c9e9d2b55c5c278f"}
Nov 02 18:15:18.568712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.568289    4476 scope.go:122] "RemoveContainer" containerID="fe676a028338c57d5e7063403678b82e9447283caaef22ee783f646037df071c"
Nov 02 18:15:18.577351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.577319    4476 generic.go:356] "Generic (PLEG): container finished" podID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerID="7b16d54a74a80cf50e46d52dd50316f89d8db80c87e7841e3b31d5983e91854e" exitCode=60
Nov 02 18:15:18.577484 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.577422    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" event={"ID":"1e25d042-df0e-4c8b-aec0-6870982714e2","Type":"ContainerDied","Data":"7b16d54a74a80cf50e46d52dd50316f89d8db80c87e7841e3b31d5983e91854e"}
Nov 02 18:15:18.578094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.577870    4476 scope.go:122] "RemoveContainer" containerID="7b16d54a74a80cf50e46d52dd50316f89d8db80c87e7841e3b31d5983e91854e"
Nov 02 18:15:18.578227 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:18.578205    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s_odh-ai-gateway-infra(1e25d042-df0e-4c8b-aec0-6870982714e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2"
Nov 02 18:15:18.580147 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.580126    4476 generic.go:356] "Generic (PLEG): container finished" podID="17979935-c344-4bad-a54d-3a47ed7b294f" containerID="18d4f6fb452cd0f65e4d9e4f7173bcefcfdfb221c46a563e7dd2ec7fd27ffef7" exitCode=60
Nov 02 18:15:18.580246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.580156    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" event={"ID":"17979935-c344-4bad-a54d-3a47ed7b294f","Type":"ContainerDied","Data":"18d4f6fb452cd0f65e4d9e4f7173bcefcfdfb221c46a563e7dd2ec7fd27ffef7"}
Nov 02 18:15:18.580457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.580437    4476 scope.go:122] "RemoveContainer" containerID="18d4f6fb452cd0f65e4d9e4f7173bcefcfdfb221c46a563e7dd2ec7fd27ffef7"
Nov 02 18:15:18.580757 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:18.580736    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm_odh-ai-gateway-infra(17979935-c344-4bad-a54d-3a47ed7b294f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" podUID="17979935-c344-4bad-a54d-3a47ed7b294f"
Nov 02 18:15:18.583531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.583508    4476 scope.go:122] "RemoveContainer" containerID="55e016a07a2503392eb980833a1daa4622959b8d17556240c6c8f3a866c1523e"
Nov 02 18:15:18.597166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.597137    4476 scope.go:122] "RemoveContainer" containerID="55d25d7a8bd48c6f9dce69f9835c3ca15962aebf052d978c7210e514518e913f"
Nov 02 18:15:18.687229 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.687199    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"]
Nov 02 18:15:18.837976 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.837947    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:18.888905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.888874    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca\") pod \"b33db761-c384-44a6-9765-0abeae673657\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") "
Nov 02 18:15:18.889122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.889103    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g\" (UniqueName: \"kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g\") pod \"b33db761-c384-44a6-9765-0abeae673657\" (UID: \"b33db761-c384-44a6-9765-0abeae673657\") "
Nov 02 18:15:18.889273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.889253    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca" pod "b33db761-c384-44a6-9765-0abeae673657" (UID: "b33db761-c384-44a6-9765-0abeae673657"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:18.889548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.889531    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33db761-c384-44a6-9765-0abeae673657-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:18.891542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.891514    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g" pod "b33db761-c384-44a6-9765-0abeae673657" (UID: "b33db761-c384-44a6-9765-0abeae673657"). InnerVolumeSpecName "kube-api-access-dtz7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:18.966567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.966533    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"]
Nov 02 18:15:18.973870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.973844    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894055-f77h4"]
Nov 02 18:15:18.990504 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:18.990470    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtz7g\" (UniqueName: \"kubernetes.io/projected/b33db761-c384-44a6-9765-0abeae673657-kube-api-access-dtz7g\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:19.383589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.383558    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" path="/var/lib/kubelet/pods/86a3bb17-4a0d-4377-b731-33bc3de59a96/volumes"
Nov 02 18:15:19.592556 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.592483    4476 generic.go:356] "Generic (PLEG): container finished" podID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerID="a641e7f6a728194eec9a7706c90f23fb9885bde60add8bf2c887622dc63ea5a7" exitCode=60
Nov 02 18:15:19.593044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.592558    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerDied","Data":"a641e7f6a728194eec9a7706c90f23fb9885bde60add8bf2c887622dc63ea5a7"}
Nov 02 18:15:19.593044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.592613    4476 scope.go:122] "RemoveContainer" containerID="6af1ae5d4d8b8a96c71751feab9ec9e520a57bb21b53cf6e23f6f170bfe4aa56"
Nov 02 18:15:19.593164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.593063    4476 scope.go:122] "RemoveContainer" containerID="a641e7f6a728194eec9a7706c90f23fb9885bde60add8bf2c887622dc63ea5a7"
Nov 02 18:15:19.593427 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:19.593399    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q_odh-ai-gateway-infra(c2d7331b-5dd1-4283-a58f-f427294e8b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30"
Nov 02 18:15:19.595137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.595115    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh" event={"ID":"b33db761-c384-44a6-9765-0abeae673657","Type":"ContainerDied","Data":"3a3a54ed35439a2b1e4b70fa121d1a4f5044bcda9965822492647c0a1dda3cdd"}
Nov 02 18:15:19.595230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.595139    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"
Nov 02 18:15:19.605943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.605922    4476 scope.go:122] "RemoveContainer" containerID="44698cc344a314df2dda4b6c985598a5278bf8407ec01de75eee4db27a872d3b"
Nov 02 18:15:19.710491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.710455    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:19.745225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.745192    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"]
Nov 02 18:15:19.820784 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.820753    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"]
Nov 02 18:15:19.824330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.824303    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894055-zr8dh"]
Nov 02 18:15:19.858441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.858356    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"]
Nov 02 18:15:19.915293 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:19.915268    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:20.004877 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.004848    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca\") pod \"17979935-c344-4bad-a54d-3a47ed7b294f\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") "
Nov 02 18:15:20.005057 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.005042    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8\" (UniqueName: \"kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8\") pod \"17979935-c344-4bad-a54d-3a47ed7b294f\" (UID: \"17979935-c344-4bad-a54d-3a47ed7b294f\") "
Nov 02 18:15:20.005200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.005175    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca" pod "17979935-c344-4bad-a54d-3a47ed7b294f" (UID: "17979935-c344-4bad-a54d-3a47ed7b294f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:20.005412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.005393    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17979935-c344-4bad-a54d-3a47ed7b294f-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:20.007352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.007267    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8" pod "17979935-c344-4bad-a54d-3a47ed7b294f" (UID: "17979935-c344-4bad-a54d-3a47ed7b294f"). InnerVolumeSpecName "kube-api-access-fgzt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:20.029319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.029283    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:20.106021 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.105987    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84\" (UniqueName: \"kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84\") pod \"1e25d042-df0e-4c8b-aec0-6870982714e2\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") "
Nov 02 18:15:20.106021 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.106028    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca\") pod \"1e25d042-df0e-4c8b-aec0-6870982714e2\" (UID: \"1e25d042-df0e-4c8b-aec0-6870982714e2\") "
Nov 02 18:15:20.106339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.106327    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzt8\" (UniqueName: \"kubernetes.io/projected/17979935-c344-4bad-a54d-3a47ed7b294f-kube-api-access-fgzt8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:20.106435 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.106411    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca" pod "1e25d042-df0e-4c8b-aec0-6870982714e2" (UID: "1e25d042-df0e-4c8b-aec0-6870982714e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:20.108327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.108308    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84" pod "1e25d042-df0e-4c8b-aec0-6870982714e2" (UID: "1e25d042-df0e-4c8b-aec0-6870982714e2"). InnerVolumeSpecName "kube-api-access-rbd84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:20.207665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.207616    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbd84\" (UniqueName: \"kubernetes.io/projected/1e25d042-df0e-4c8b-aec0-6870982714e2-kube-api-access-rbd84\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:20.207665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.207665    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e25d042-df0e-4c8b-aec0-6870982714e2-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:20.603667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.603482    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"
Nov 02 18:15:20.603667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.603488    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s" event={"ID":"1e25d042-df0e-4c8b-aec0-6870982714e2","Type":"ContainerDied","Data":"355930703b0f9ed696af60f550814c6b875eefc92e0ea89f593b6355fbdaf29a"}
Nov 02 18:15:20.603667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.603659    4476 scope.go:122] "RemoveContainer" containerID="7b16d54a74a80cf50e46d52dd50316f89d8db80c87e7841e3b31d5983e91854e"
Nov 02 18:15:20.630175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.605110    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"
Nov 02 18:15:20.630175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.605140    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm" event={"ID":"17979935-c344-4bad-a54d-3a47ed7b294f","Type":"ContainerDied","Data":"59c8740cf853765430bf2d4b6675c89c27b90b8f4df014ae094a290f3eaca2b3"}
Nov 02 18:15:20.630175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.619269    4476 scope.go:122] "RemoveContainer" containerID="18d4f6fb452cd0f65e4d9e4f7173bcefcfdfb221c46a563e7dd2ec7fd27ffef7"
Nov 02 18:15:20.717122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.717091    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"]
Nov 02 18:15:20.768160 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.768128    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"]
Nov 02 18:15:20.771817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.771794    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894055-nkxkm"]
Nov 02 18:15:20.881091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.881053    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"]
Nov 02 18:15:20.883693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.883669    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:20.884532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:20.884508    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894055-7nc8s"]
Nov 02 18:15:21.016739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.016706    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk\" (UniqueName: \"kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk\") pod \"c2d7331b-5dd1-4283-a58f-f427294e8b30\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") "
Nov 02 18:15:21.016969 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.016828    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca\") pod \"c2d7331b-5dd1-4283-a58f-f427294e8b30\" (UID: \"c2d7331b-5dd1-4283-a58f-f427294e8b30\") "
Nov 02 18:15:21.017205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.017182    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca" pod "c2d7331b-5dd1-4283-a58f-f427294e8b30" (UID: "c2d7331b-5dd1-4283-a58f-f427294e8b30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:21.019309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.019279    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk" pod "c2d7331b-5dd1-4283-a58f-f427294e8b30" (UID: "c2d7331b-5dd1-4283-a58f-f427294e8b30"). InnerVolumeSpecName "kube-api-access-xmhwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:21.118214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.118162    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhwk\" (UniqueName: \"kubernetes.io/projected/c2d7331b-5dd1-4283-a58f-f427294e8b30-kube-api-access-xmhwk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:21.118214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.118192    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d7331b-5dd1-4283-a58f-f427294e8b30-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:21.385991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.385894    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" path="/var/lib/kubelet/pods/17979935-c344-4bad-a54d-3a47ed7b294f/volumes"
Nov 02 18:15:21.386423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.386401    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" path="/var/lib/kubelet/pods/1e25d042-df0e-4c8b-aec0-6870982714e2/volumes"
Nov 02 18:15:21.386871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.386852    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33db761-c384-44a6-9765-0abeae673657" path="/var/lib/kubelet/pods/b33db761-c384-44a6-9765-0abeae673657/volumes"
Nov 02 18:15:21.619207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.619176    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"
Nov 02 18:15:21.619731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.619211    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q" event={"ID":"c2d7331b-5dd1-4283-a58f-f427294e8b30","Type":"ContainerDied","Data":"0f492f5a2c5c2d28ee37f6e928d8df17b0cb83658abab02a521452e8b6ddeb79"}
Nov 02 18:15:21.619731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.619271    4476 scope.go:122] "RemoveContainer" containerID="a641e7f6a728194eec9a7706c90f23fb9885bde60add8bf2c887622dc63ea5a7"
Nov 02 18:15:21.783089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.782992    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"]
Nov 02 18:15:21.786339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:21.786316    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894055-gzs9q"]
Nov 02 18:15:22.056123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:22.055809    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"]
Nov 02 18:15:22.128947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:22.128908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:23.384106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:23.384067    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" path="/var/lib/kubelet/pods/c2d7331b-5dd1-4283-a58f-f427294e8b30/volumes"
Nov 02 18:15:24.498801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:24.498771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:25.610061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:25.610028    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"]
Nov 02 18:15:25.610518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:25.610219    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" containerID="cri-o://c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec" gracePeriod=30
Nov 02 18:15:29.060902 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:29.060857    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:15:29.266283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:29.266244    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:29.476199 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:29.476147    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:15:29.476408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:29.476214    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:15:30.889501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:30.889436    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:15:31.034487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034443    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.034720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034512    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.034720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034562    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.034720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034579    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.034720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034645    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.034720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034713    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.035044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034797    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.035044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034825    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:31.035044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034834    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.035044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.034907    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy\") pod \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\" (UID: \"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0\") "
Nov 02 18:15:31.035044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035037    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:31.035321 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035231    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:31.035661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035434    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:31.035661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035538    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.035661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035559    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.035661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035574    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.035661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035587    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.035661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.035644    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:31.037941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.037882    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:31.037941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.037907    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "kube-api-access-zwq29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:31.038558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.038536    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:31.039801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.039774    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo" pod "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" (UID: "55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:15:31.136394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.136363    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.136394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.136391    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwq29\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-kube-api-access-zwq29\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.136394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.136403    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.136670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.136412    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.136670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.136420    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:31.702572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.702472    4476 generic.go:356] "Generic (PLEG): container finished" podID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerID="c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec" exitCode=0
Nov 02 18:15:31.702572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.702531    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"
Nov 02 18:15:31.702572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.702556    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" event={"ID":"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0","Type":"ContainerDied","Data":"c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec"}
Nov 02 18:15:31.702876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.702617    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c" event={"ID":"55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0","Type":"ContainerDied","Data":"a6b1e27003b185122d727e3bd19f1a98d0cf6d03d141f99917ce1620308a9d17"}
Nov 02 18:15:31.702876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.702663    4476 scope.go:122] "RemoveContainer" containerID="c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec"
Nov 02 18:15:31.714071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.714056    4476 scope.go:122] "RemoveContainer" containerID="c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec"
Nov 02 18:15:31.714325 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:31.714309    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec\": container with ID starting with c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec not found: ID does not exist" containerID="c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec"
Nov 02 18:15:31.714368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.714332    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec"} err="failed to get container status \"c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec\": rpc error: code = NotFound desc = could not find container \"c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec\": container with ID starting with c41fecc86d87908650647395e5e31de7bdb1a460e4de684dc352a4769e2a88ec not found: ID does not exist"
Nov 02 18:15:31.868605 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.868571    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"]
Nov 02 18:15:31.915939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:31.915901    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-klf9c"]
Nov 02 18:15:33.384574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:33.384539    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" path="/var/lib/kubelet/pods/55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0/volumes"
Nov 02 18:15:33.954963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:33.954930    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:37.080504 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.080460    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l" podUID="2a961254-cbfe-4979-b041-12879d6ab4fa" containerName="main" containerID="cri-o://637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769" gracePeriod=45
Nov 02 18:15:37.348555 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.348483    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:15:37.407448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407417    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir\") pod \"2a961254-cbfe-4979-b041-12879d6ab4fa\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") "
Nov 02 18:15:37.407670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407456    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache\") pod \"2a961254-cbfe-4979-b041-12879d6ab4fa\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") "
Nov 02 18:15:37.407670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407482    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm\") pod \"2a961254-cbfe-4979-b041-12879d6ab4fa\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") "
Nov 02 18:15:37.407670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407520    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs\") pod \"2a961254-cbfe-4979-b041-12879d6ab4fa\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") "
Nov 02 18:15:37.407670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407551    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl\" (UniqueName: \"kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl\") pod \"2a961254-cbfe-4979-b041-12879d6ab4fa\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") "
Nov 02 18:15:37.407670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407578    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home\") pod \"2a961254-cbfe-4979-b041-12879d6ab4fa\" (UID: \"2a961254-cbfe-4979-b041-12879d6ab4fa\") "
Nov 02 18:15:37.407983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407755    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir" pod "2a961254-cbfe-4979-b041-12879d6ab4fa" (UID: "2a961254-cbfe-4979-b041-12879d6ab4fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:37.407983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.407790    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache" pod "2a961254-cbfe-4979-b041-12879d6ab4fa" (UID: "2a961254-cbfe-4979-b041-12879d6ab4fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:37.408095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.408040    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home" pod "2a961254-cbfe-4979-b041-12879d6ab4fa" (UID: "2a961254-cbfe-4979-b041-12879d6ab4fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:37.408250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.408227    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:37.408250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.408252    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:37.408456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.408265    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:37.410075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.410048    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm" pod "2a961254-cbfe-4979-b041-12879d6ab4fa" (UID: "2a961254-cbfe-4979-b041-12879d6ab4fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:37.410376 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.410349    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs" pod "2a961254-cbfe-4979-b041-12879d6ab4fa" (UID: "2a961254-cbfe-4979-b041-12879d6ab4fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:37.410462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.410386    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl" pod "2a961254-cbfe-4979-b041-12879d6ab4fa" (UID: "2a961254-cbfe-4979-b041-12879d6ab4fa"). InnerVolumeSpecName "kube-api-access-lmpgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:37.509790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.509747    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a961254-cbfe-4979-b041-12879d6ab4fa-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:37.509790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.509776    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a961254-cbfe-4979-b041-12879d6ab4fa-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:37.509790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.509786    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmpgl\" (UniqueName: \"kubernetes.io/projected/2a961254-cbfe-4979-b041-12879d6ab4fa-kube-api-access-lmpgl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:15:37.745801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.745767    4476 generic.go:356] "Generic (PLEG): container finished" podID="2a961254-cbfe-4979-b041-12879d6ab4fa" containerID="637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769" exitCode=0
Nov 02 18:15:37.746005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.745817    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"
Nov 02 18:15:37.746005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.745860    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l" event={"ID":"2a961254-cbfe-4979-b041-12879d6ab4fa","Type":"ContainerDied","Data":"637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769"}
Nov 02 18:15:37.746005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.745911    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l" event={"ID":"2a961254-cbfe-4979-b041-12879d6ab4fa","Type":"ContainerDied","Data":"488ba740e2fd19ac561610e46de49ae425bb7618924c3ab0a6f53bf1bb51ea01"}
Nov 02 18:15:37.746005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.745932    4476 scope.go:122] "RemoveContainer" containerID="637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769"
Nov 02 18:15:37.759997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.758935    4476 scope.go:122] "RemoveContainer" containerID="637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769"
Nov 02 18:15:37.760224 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:37.760196    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769\": container with ID starting with 637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769 not found: ID does not exist" containerID="637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769"
Nov 02 18:15:37.760334 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.760232    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769"} err="failed to get container status \"637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769\": rpc error: code = NotFound desc = could not find container \"637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769\": container with ID starting with 637477e31d95bbb87b8cf4993ebb2f37f34e7a8fb7ed61f9da0b9a62ad14d769 not found: ID does not exist"
Nov 02 18:15:37.908900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.908868    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"]
Nov 02 18:15:37.960236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:37.960197    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/auth-test-model-c82472-kserve-5dbb4456c7-7zh5l"]
Nov 02 18:15:38.711109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:38.711074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:39.386354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:39.386320    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a961254-cbfe-4979-b041-12879d6ab4fa" path="/var/lib/kubelet/pods/2a961254-cbfe-4979-b041-12879d6ab4fa/volumes"
Nov 02 18:15:43.472089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:43.471996    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:48.319299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:48.319265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:53.062165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:53.062127    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:57.751229 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:57.751193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:57.933467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:15:57.933431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:59.061100 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:15:59.061054    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:16:02.336753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:02.336668    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:02.555207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:02.555170    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:02.755304 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:02.755268    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:07.211216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:07.211166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:11.963362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:11.963326    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:15.569951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.569918    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"]
Nov 02 18:16:15.570421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570409    4476 cpu_manager.go:414] "containerMap: removing container" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:15.570421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570420    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570432    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570437    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570445    4476 cpu_manager.go:414] "containerMap: removing container" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570450    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570460    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570465    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570471    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2a961254-cbfe-4979-b041-12879d6ab4fa" containerName="main"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570476    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a961254-cbfe-4979-b041-12879d6ab4fa" containerName="main"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570487    4476 cpu_manager.go:414] "containerMap: removing container" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:15.570495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570491    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570500    4476 cpu_manager.go:414] "containerMap: removing container" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570504    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570512    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570517    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570525    4476 cpu_manager.go:414] "containerMap: removing container" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570530    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570536    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570539    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570545    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570549    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570558    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570562    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570571    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570575    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570582    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570586    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570592    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570597    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570607    4476 cpu_manager.go:414] "containerMap: removing container" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570611    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570735    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570744    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570751    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570756    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570761    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570766    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570771    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d7331b-5dd1-4283-a58f-f427294e8b30" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570776    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570783    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570789    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570794    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570798    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e971b0-0a6b-4e39-b0a9-f2cf7192e6a0" containerName="istio-proxy"
Nov 02 18:16:15.570841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.570804    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a961254-cbfe-4979-b041-12879d6ab4fa" containerName="main"
Nov 02 18:16:15.575518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.575502    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.629880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.629847    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"]
Nov 02 18:16:15.674502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.674474    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:16:15.674694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.674553    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:16:15.683150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.683119    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.683253 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.683239    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.683299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.683277    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzt7\" (UniqueName: \"kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.784258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.784217    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.784439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.784322    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.784439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.784372    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzt7\" (UniqueName: \"kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.785040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.785018    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.786672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.786651    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.850508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.850440    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzt7\" (UniqueName: \"kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7\") pod \"maas-api-revoke-sub-7181d7e2-t88ss\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:15.889499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:15.889468    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:16.273215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:16.273187    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"]
Nov 02 18:16:16.275178 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:16:16.275147    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2e934a_ec45_436c_aa46_1fde5b13ac2d.slice/crio-ce1b03f766a747b46288409598d4c11df2c84eb57dc43e7c48940a97daecf65d WatchSource:0}: Error finding container ce1b03f766a747b46288409598d4c11df2c84eb57dc43e7c48940a97daecf65d: Status 404 returned error can't find the container with id ce1b03f766a747b46288409598d4c11df2c84eb57dc43e7c48940a97daecf65d
Nov 02 18:16:16.684738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:16.684681    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:17.045880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:17.045839    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss" event={"ID":"5a2e934a-ec45-436c-aa46-1fde5b13ac2d","Type":"ContainerStarted","Data":"f760925d7bac74d74b67d8038a555a9d1cacaf2c84ac20cfa4a78046a18cefb2"}
Nov 02 18:16:17.046067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:17.045906    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss" event={"ID":"5a2e934a-ec45-436c-aa46-1fde5b13ac2d","Type":"ContainerStarted","Data":"ce1b03f766a747b46288409598d4c11df2c84eb57dc43e7c48940a97daecf65d"}
Nov 02 18:16:17.164290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:17.164232    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss" podStartSLOduration=2.164216419 podStartE2EDuration="2.164216419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:17.160239636 +0000 UTC m=+3208.000845574" watchObservedRunningTime="2026-11-02 18:16:17.164216419 +0000 UTC m=+3208.004822353"
Nov 02 18:16:18.054341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:18.054299    4476 generic.go:356] "Generic (PLEG): container finished" podID="5a2e934a-ec45-436c-aa46-1fde5b13ac2d" containerID="f760925d7bac74d74b67d8038a555a9d1cacaf2c84ac20cfa4a78046a18cefb2" exitCode=0
Nov 02 18:16:18.054810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:18.054383    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss" event={"ID":"5a2e934a-ec45-436c-aa46-1fde5b13ac2d","Type":"ContainerDied","Data":"f760925d7bac74d74b67d8038a555a9d1cacaf2c84ac20cfa4a78046a18cefb2"}
Nov 02 18:16:19.204446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.204423    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:19.325411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.325325    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca\") pod \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") "
Nov 02 18:16:19.325589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.325416    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7\" (UniqueName: \"kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7\") pod \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") "
Nov 02 18:16:19.325695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.325642    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca\") pod \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\" (UID: \"5a2e934a-ec45-436c-aa46-1fde5b13ac2d\") "
Nov 02 18:16:19.325760 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.325739    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca" pod "5a2e934a-ec45-436c-aa46-1fde5b13ac2d" (UID: "5a2e934a-ec45-436c-aa46-1fde5b13ac2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:19.326050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.326031    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:19.327743 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.327716    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7" pod "5a2e934a-ec45-436c-aa46-1fde5b13ac2d" (UID: "5a2e934a-ec45-436c-aa46-1fde5b13ac2d"). InnerVolumeSpecName "kube-api-access-hfzt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:19.328027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.328007    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca" pod "5a2e934a-ec45-436c-aa46-1fde5b13ac2d" (UID: "5a2e934a-ec45-436c-aa46-1fde5b13ac2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:16:19.426780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.426746    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:19.426780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:19.426779    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzt7\" (UniqueName: \"kubernetes.io/projected/5a2e934a-ec45-436c-aa46-1fde5b13ac2d-kube-api-access-hfzt7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:20.069598 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:20.069561    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"
Nov 02 18:16:20.069598 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:20.069575    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss" event={"ID":"5a2e934a-ec45-436c-aa46-1fde5b13ac2d","Type":"ContainerDied","Data":"ce1b03f766a747b46288409598d4c11df2c84eb57dc43e7c48940a97daecf65d"}
Nov 02 18:16:20.069856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:20.069682    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce1b03f766a747b46288409598d4c11df2c84eb57dc43e7c48940a97daecf65d"
Nov 02 18:16:21.303455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:21.303423    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:21.489257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:21.489224    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:26.079220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.079184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:26.309744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.309707    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"]
Nov 02 18:16:26.310242 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310229    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5a2e934a-ec45-436c-aa46-1fde5b13ac2d" containerName="revoke-keys"
Nov 02 18:16:26.310298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310243    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2e934a-ec45-436c-aa46-1fde5b13ac2d" containerName="revoke-keys"
Nov 02 18:16:26.310298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310255    4476 cpu_manager.go:414] "containerMap: removing container" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:26.310298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310260    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:26.310403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310330    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a3bb17-4a0d-4377-b731-33bc3de59a96" containerName="cleanup"
Nov 02 18:16:26.310403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310339    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e25d042-df0e-4c8b-aec0-6870982714e2" containerName="cleanup"
Nov 02 18:16:26.310403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310348    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2e934a-ec45-436c-aa46-1fde5b13ac2d" containerName="revoke-keys"
Nov 02 18:16:26.310403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310359    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="17979935-c344-4bad-a54d-3a47ed7b294f" containerName="cleanup"
Nov 02 18:16:26.310403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.310366    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33db761-c384-44a6-9765-0abeae673657" containerName="cleanup"
Nov 02 18:16:26.313754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.313734    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.374549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.374513    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"]
Nov 02 18:16:26.404657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.404592    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.404837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.404793    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqqj\" (UniqueName: \"kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.404923 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.404835    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.418449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.418428    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:16:26.418566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.418466    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:16:26.506439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.506396    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqqj\" (UniqueName: \"kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.506691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.506440    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.506691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.506670    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.507413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.507387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.509157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.509139    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.563111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.563079    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqqj\" (UniqueName: \"kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj\") pod \"maas-api-revoke-sub-268c9a78-d2b55\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:26.626890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:26.626815    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:27.006826 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:27.006789    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"]
Nov 02 18:16:27.007945 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:16:27.007913    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d70b6c5_5e72_4498_a33a_08b6a7a925b8.slice/crio-e4c8ab6cb568bec152d6ba1456a1f6c29208f6a5d3cfe15fac311112c3108f5d WatchSource:0}: Error finding container e4c8ab6cb568bec152d6ba1456a1f6c29208f6a5d3cfe15fac311112c3108f5d: Status 404 returned error can't find the container with id e4c8ab6cb568bec152d6ba1456a1f6c29208f6a5d3cfe15fac311112c3108f5d
Nov 02 18:16:27.131062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:27.131027    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55" event={"ID":"4d70b6c5-5e72-4498-a33a-08b6a7a925b8","Type":"ContainerStarted","Data":"e4c8ab6cb568bec152d6ba1456a1f6c29208f6a5d3cfe15fac311112c3108f5d"}
Nov 02 18:16:28.140337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:28.140294    4476 generic.go:356] "Generic (PLEG): container finished" podID="4d70b6c5-5e72-4498-a33a-08b6a7a925b8" containerID="09cd49c6934cc37562128711635202df259225d43764ecb90b332f8589af0309" exitCode=0
Nov 02 18:16:28.140859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:28.140373    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55" event={"ID":"4d70b6c5-5e72-4498-a33a-08b6a7a925b8","Type":"ContainerDied","Data":"09cd49c6934cc37562128711635202df259225d43764ecb90b332f8589af0309"}
Nov 02 18:16:29.059719 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:16:29.059680    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:16:29.338263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.338238    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:29.437326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.437286    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca\") pod \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") "
Nov 02 18:16:29.437499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.437479    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca\") pod \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") "
Nov 02 18:16:29.437557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.437538    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj\" (UniqueName: \"kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj\") pod \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\" (UID: \"4d70b6c5-5e72-4498-a33a-08b6a7a925b8\") "
Nov 02 18:16:29.437740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.437709    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca" pod "4d70b6c5-5e72-4498-a33a-08b6a7a925b8" (UID: "4d70b6c5-5e72-4498-a33a-08b6a7a925b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:29.438616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.438062    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:29.440009 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.439988    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj" pod "4d70b6c5-5e72-4498-a33a-08b6a7a925b8" (UID: "4d70b6c5-5e72-4498-a33a-08b6a7a925b8"). InnerVolumeSpecName "kube-api-access-2nqqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:29.440288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.440267    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca" pod "4d70b6c5-5e72-4498-a33a-08b6a7a925b8" (UID: "4d70b6c5-5e72-4498-a33a-08b6a7a925b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:16:29.539506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.539457    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:29.539506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:29.539499    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqqj\" (UniqueName: \"kubernetes.io/projected/4d70b6c5-5e72-4498-a33a-08b6a7a925b8-kube-api-access-2nqqj\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:16:30.158441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:30.158404    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55" event={"ID":"4d70b6c5-5e72-4498-a33a-08b6a7a925b8","Type":"ContainerDied","Data":"e4c8ab6cb568bec152d6ba1456a1f6c29208f6a5d3cfe15fac311112c3108f5d"}
Nov 02 18:16:30.158441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:30.158428    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"
Nov 02 18:16:30.158715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:30.158468    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c8ab6cb568bec152d6ba1456a1f6c29208f6a5d3cfe15fac311112c3108f5d"
Nov 02 18:16:30.781821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:30.781788    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:35.661169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:35.661136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:36.879298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:36.879263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:37.954544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:37.954500    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:40.747419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:40.747382    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:41.235292 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.235259    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"]
Nov 02 18:16:41.235873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.235845    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4d70b6c5-5e72-4498-a33a-08b6a7a925b8" containerName="revoke-keys"
Nov 02 18:16:41.235873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.235863    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d70b6c5-5e72-4498-a33a-08b6a7a925b8" containerName="revoke-keys"
Nov 02 18:16:41.236115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.235993    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d70b6c5-5e72-4498-a33a-08b6a7a925b8" containerName="revoke-keys"
Nov 02 18:16:41.239665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.239610    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.295216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.295186    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"]
Nov 02 18:16:41.340528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.340496    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3dc055\"/\"openshift-service-ca.crt\""
Nov 02 18:16:41.340528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.340512    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3dc055\"/\"kube-root-ca.crt\""
Nov 02 18:16:41.340755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.340599    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3dc055\"/\"default-dockercfg-fd6t4\""
Nov 02 18:16:41.341297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.341280    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3dc055\"/\"explicit-model-3dc055-kserve-self-signed-certs\""
Nov 02 18:16:41.378460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.378069    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.378460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.378117    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.378460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.378157    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5jl\" (UniqueName: \"kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.378460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.378239    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.378460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.378288    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.378460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.378328    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.382396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.382314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:41.479567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.479535    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.479828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.479588    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.479828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.479758    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.479828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.479795    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.480013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.479831    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5jl\" (UniqueName: \"kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.480013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.479905    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.480425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.480397    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.480579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.480476    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.480579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.480518    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.482439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.482418    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.482608 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.482590    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.536516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.536426    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5jl\" (UniqueName: \"kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl\") pod \"explicit-model-3dc055-kserve-7797b57959-q8bzv\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") " pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.552714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.552679    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:41.947941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:41.947908    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"]
Nov 02 18:16:41.949188 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:16:41.949164    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ef0aa2b_2508_4887_9a76_4d1177ecf16e.slice/crio-57b6220565109e8e0ebedd754117b13f4c01aa4779f2dcb70c8f6aeafbbeaa2b WatchSource:0}: Error finding container 57b6220565109e8e0ebedd754117b13f4c01aa4779f2dcb70c8f6aeafbbeaa2b: Status 404 returned error can't find the container with id 57b6220565109e8e0ebedd754117b13f4c01aa4779f2dcb70c8f6aeafbbeaa2b
Nov 02 18:16:42.251097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:42.250986    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv" event={"ID":"3ef0aa2b-2508-4887-9a76-4d1177ecf16e","Type":"ContainerStarted","Data":"612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932"}
Nov 02 18:16:42.251097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:42.251054    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv" event={"ID":"3ef0aa2b-2508-4887-9a76-4d1177ecf16e","Type":"ContainerStarted","Data":"57b6220565109e8e0ebedd754117b13f4c01aa4779f2dcb70c8f6aeafbbeaa2b"}
Nov 02 18:16:42.367317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:42.367258    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv" podStartSLOduration=1.3672419740000001 podStartE2EDuration="1.367241974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:42.364116174 +0000 UTC m=+3233.204722120" watchObservedRunningTime="2026-11-02 18:16:42.367241974 +0000 UTC m=+3233.207847909"
Nov 02 18:16:42.553206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:42.553120    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:45.084283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:45.084247    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:45.272733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:45.272696    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:45.561017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:45.560954    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:50.251950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:50.251901    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:51.553675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:51.553639    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:51.567280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:51.567248    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:52.348388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:52.348360    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:16:54.920141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:54.920105    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:55.121463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:55.121428    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:58.219313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:58.219284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:59.063425 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:16:59.063383    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:16:59.776567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:16:59.776512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:04.412964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:04.412931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:08.171190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:08.171155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:09.172403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:09.172303    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:13.826768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:13.826676    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:18.580672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:18.580602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:20.825347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:20.825295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:23.381958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:23.381917    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:25.651619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:25.651536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:25.768024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:25.767991    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"]
Nov 02 18:17:25.886355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:25.886325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:28.407385 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:28.407345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:29.060400 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:29.060364    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:17:33.105396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:33.105298    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:36.332499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.332459    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"]
Nov 02 18:17:36.336518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.336501    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.422299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.422259    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67wl\" (UniqueName: \"kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.422479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.422452    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.422532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.422504    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.441058 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.441026    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:17:36.441205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.441079    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:17:36.444777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.444752    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"]
Nov 02 18:17:36.523456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.523422    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.523456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.523463    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.523763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.523540    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67wl\" (UniqueName: \"kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.524235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.524213    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.526145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.526118    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.580474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.580444    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67wl\" (UniqueName: \"kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl\") pod \"maas-api-revoke-sub-17af1101-p52wl\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:36.651971 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:36.651933    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:37.050108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:37.050081    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"]
Nov 02 18:17:37.052428 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:17:37.052399    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872dfecd_04fe_4472_a211_7b50b921dcb1.slice/crio-1d031da0e803c48820f05a30331cdda60c6db5ec7d4908d45710fbee4ceeb7f3 WatchSource:0}: Error finding container 1d031da0e803c48820f05a30331cdda60c6db5ec7d4908d45710fbee4ceeb7f3: Status 404 returned error can't find the container with id 1d031da0e803c48820f05a30331cdda60c6db5ec7d4908d45710fbee4ceeb7f3
Nov 02 18:17:37.692818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:37.692765    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl" event={"ID":"872dfecd-04fe-4472-a211-7b50b921dcb1","Type":"ContainerStarted","Data":"1a3c175064c19b17ef31b534ba6455b2e51c6ecd09b1b63de6bea3bd3559551c"}
Nov 02 18:17:37.693210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:37.692838    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl" event={"ID":"872dfecd-04fe-4472-a211-7b50b921dcb1","Type":"ContainerStarted","Data":"1d031da0e803c48820f05a30331cdda60c6db5ec7d4908d45710fbee4ceeb7f3"}
Nov 02 18:17:37.808684 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:37.808566    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl" podStartSLOduration=1.808552696 podStartE2EDuration="1.808552696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:37.805032898 +0000 UTC m=+3288.645638833" watchObservedRunningTime="2026-11-02 18:17:37.808552696 +0000 UTC m=+3288.649158629"
Nov 02 18:17:37.857438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:37.857372    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:38.702076 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:38.702043    4476 generic.go:356] "Generic (PLEG): container finished" podID="872dfecd-04fe-4472-a211-7b50b921dcb1" containerID="1a3c175064c19b17ef31b534ba6455b2e51c6ecd09b1b63de6bea3bd3559551c" exitCode=0
Nov 02 18:17:38.702525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:38.702131    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl" event={"ID":"872dfecd-04fe-4472-a211-7b50b921dcb1","Type":"ContainerDied","Data":"1a3c175064c19b17ef31b534ba6455b2e51c6ecd09b1b63de6bea3bd3559551c"}
Nov 02 18:17:39.846681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.846649    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:39.967076 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.966995    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl\" (UniqueName: \"kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl\") pod \"872dfecd-04fe-4472-a211-7b50b921dcb1\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") "
Nov 02 18:17:39.967076 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.967045    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca\") pod \"872dfecd-04fe-4472-a211-7b50b921dcb1\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") "
Nov 02 18:17:39.967315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.967171    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca\") pod \"872dfecd-04fe-4472-a211-7b50b921dcb1\" (UID: \"872dfecd-04fe-4472-a211-7b50b921dcb1\") "
Nov 02 18:17:39.967522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.967494    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca" pod "872dfecd-04fe-4472-a211-7b50b921dcb1" (UID: "872dfecd-04fe-4472-a211-7b50b921dcb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:17:39.967701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.967680    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:39.969895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.969866    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl" pod "872dfecd-04fe-4472-a211-7b50b921dcb1" (UID: "872dfecd-04fe-4472-a211-7b50b921dcb1"). InnerVolumeSpecName "kube-api-access-p67wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:39.970100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:39.970082    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca" pod "872dfecd-04fe-4472-a211-7b50b921dcb1" (UID: "872dfecd-04fe-4472-a211-7b50b921dcb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:40.068720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:40.068686    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67wl\" (UniqueName: \"kubernetes.io/projected/872dfecd-04fe-4472-a211-7b50b921dcb1-kube-api-access-p67wl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:40.068720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:40.068717    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872dfecd-04fe-4472-a211-7b50b921dcb1-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:40.719162 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:40.719128    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"
Nov 02 18:17:40.719377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:40.719134    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl" event={"ID":"872dfecd-04fe-4472-a211-7b50b921dcb1","Type":"ContainerDied","Data":"1d031da0e803c48820f05a30331cdda60c6db5ec7d4908d45710fbee4ceeb7f3"}
Nov 02 18:17:40.719453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:40.719395    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d031da0e803c48820f05a30331cdda60c6db5ec7d4908d45710fbee4ceeb7f3"
Nov 02 18:17:40.789397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:40.789340    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv" podUID="3ef0aa2b-2508-4887-9a76-4d1177ecf16e" containerName="main" containerID="cri-o://612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932" gracePeriod=45
Nov 02 18:17:41.081219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.081195    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:17:41.182208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182178    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home\") pod \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") "
Nov 02 18:17:41.182422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182235    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl\" (UniqueName: \"kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl\") pod \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") "
Nov 02 18:17:41.182422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182254    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache\") pod \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") "
Nov 02 18:17:41.182422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182272    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs\") pod \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") "
Nov 02 18:17:41.182422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182320    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm\") pod \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") "
Nov 02 18:17:41.182422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182337    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir\") pod \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\" (UID: \"3ef0aa2b-2508-4887-9a76-4d1177ecf16e\") "
Nov 02 18:17:41.182763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182529    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache" pod "3ef0aa2b-2508-4887-9a76-4d1177ecf16e" (UID: "3ef0aa2b-2508-4887-9a76-4d1177ecf16e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:41.182763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182541    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home" pod "3ef0aa2b-2508-4887-9a76-4d1177ecf16e" (UID: "3ef0aa2b-2508-4887-9a76-4d1177ecf16e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:41.182883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182788    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir" pod "3ef0aa2b-2508-4887-9a76-4d1177ecf16e" (UID: "3ef0aa2b-2508-4887-9a76-4d1177ecf16e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:41.182883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182819    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:41.182883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.182840    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:41.185285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.185252    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm" pod "3ef0aa2b-2508-4887-9a76-4d1177ecf16e" (UID: "3ef0aa2b-2508-4887-9a76-4d1177ecf16e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:41.185401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.185325    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs" pod "3ef0aa2b-2508-4887-9a76-4d1177ecf16e" (UID: "3ef0aa2b-2508-4887-9a76-4d1177ecf16e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:41.186935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.186913    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl" pod "3ef0aa2b-2508-4887-9a76-4d1177ecf16e" (UID: "3ef0aa2b-2508-4887-9a76-4d1177ecf16e"). InnerVolumeSpecName "kube-api-access-hd5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:41.284070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.283984    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5jl\" (UniqueName: \"kubernetes.io/projected/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-kube-api-access-hd5jl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:41.284070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.284014    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:41.284070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.284025    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:41.284070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.284054    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef0aa2b-2508-4887-9a76-4d1177ecf16e-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:41.726746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.726714    4476 generic.go:356] "Generic (PLEG): container finished" podID="3ef0aa2b-2508-4887-9a76-4d1177ecf16e" containerID="612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932" exitCode=0
Nov 02 18:17:41.726932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.726766    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"
Nov 02 18:17:41.726932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.726807    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv" event={"ID":"3ef0aa2b-2508-4887-9a76-4d1177ecf16e","Type":"ContainerDied","Data":"612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932"}
Nov 02 18:17:41.726932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.726855    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv" event={"ID":"3ef0aa2b-2508-4887-9a76-4d1177ecf16e","Type":"ContainerDied","Data":"57b6220565109e8e0ebedd754117b13f4c01aa4779f2dcb70c8f6aeafbbeaa2b"}
Nov 02 18:17:41.726932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.726877    4476 scope.go:122] "RemoveContainer" containerID="612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932"
Nov 02 18:17:41.737881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.737859    4476 scope.go:122] "RemoveContainer" containerID="612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932"
Nov 02 18:17:41.738177 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:41.738155    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932\": container with ID starting with 612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932 not found: ID does not exist" containerID="612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932"
Nov 02 18:17:41.738249 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.738190    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932"} err="failed to get container status \"612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932\": rpc error: code = NotFound desc = could not find container \"612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932\": container with ID starting with 612644a7c518ba9148d76e6201b9bdfb0269c85df8ec04abd72c354cf7b9f932 not found: ID does not exist"
Nov 02 18:17:41.897387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.897347    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"]
Nov 02 18:17:41.944766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.944731    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-3dc055/explicit-model-3dc055-kserve-7797b57959-q8bzv"]
Nov 02 18:17:41.948029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.948009    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"]
Nov 02 18:17:41.999221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:41.999129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:42.528371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:42.528338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:43.027063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:43.027015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:43.384672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:43.384618    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef0aa2b-2508-4887-9a76-4d1177ecf16e" path="/var/lib/kubelet/pods/3ef0aa2b-2508-4887-9a76-4d1177ecf16e/volumes"
Nov 02 18:17:44.147878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.147843    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"]
Nov 02 18:17:44.148374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.148361    4476 cpu_manager.go:414] "containerMap: removing container" podUID="872dfecd-04fe-4472-a211-7b50b921dcb1" containerName="revoke-keys"
Nov 02 18:17:44.148420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.148376    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872dfecd-04fe-4472-a211-7b50b921dcb1" containerName="revoke-keys"
Nov 02 18:17:44.148420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.148393    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3ef0aa2b-2508-4887-9a76-4d1177ecf16e" containerName="main"
Nov 02 18:17:44.148420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.148398    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef0aa2b-2508-4887-9a76-4d1177ecf16e" containerName="main"
Nov 02 18:17:44.148524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.148471    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="872dfecd-04fe-4472-a211-7b50b921dcb1" containerName="revoke-keys"
Nov 02 18:17:44.148524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.148482    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef0aa2b-2508-4887-9a76-4d1177ecf16e" containerName="main"
Nov 02 18:17:44.153105 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.153084    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.209025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.208990    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"]
Nov 02 18:17:44.253083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.253046    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:17:44.253083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.253079    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:17:44.327040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.327004    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.327040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.327039    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjxs\" (UniqueName: \"kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.327250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.327065    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.428241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.428156    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.428241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.428193    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjxs\" (UniqueName: \"kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.428241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.428233    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.429007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.428985    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.430717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.430696    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.486000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.485966    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjxs\" (UniqueName: \"kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs\") pod \"maas-api-revoke-sub-80f2595e-jz6vd\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:44.766973 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:44.766894    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:45.150733 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:17:45.150701    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6378f1c0_7559_4bb1_85aa_262a96899475.slice/crio-ee19db0dd86d77eb76f2af18c0377152209d79c08aa9d142c5e222225e1d6f4f WatchSource:0}: Error finding container ee19db0dd86d77eb76f2af18c0377152209d79c08aa9d142c5e222225e1d6f4f: Status 404 returned error can't find the container with id ee19db0dd86d77eb76f2af18c0377152209d79c08aa9d142c5e222225e1d6f4f
Nov 02 18:17:45.151063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:45.150825    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"]
Nov 02 18:17:45.759697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:45.759666    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd" event={"ID":"6378f1c0-7559-4bb1-85aa-262a96899475","Type":"ContainerStarted","Data":"2da6f2fd091ab75c358f89260e7069ef12f5c72535203063d4000756d196bccc"}
Nov 02 18:17:45.759902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:45.759718    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd" event={"ID":"6378f1c0-7559-4bb1-85aa-262a96899475","Type":"ContainerStarted","Data":"ee19db0dd86d77eb76f2af18c0377152209d79c08aa9d142c5e222225e1d6f4f"}
Nov 02 18:17:45.882558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:45.882497    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd" podStartSLOduration=1.8824785670000002 podStartE2EDuration="1.882478567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:45.879731744 +0000 UTC m=+3296.720337682" watchObservedRunningTime="2026-11-02 18:17:45.882478567 +0000 UTC m=+3296.723084504"
Nov 02 18:17:46.769373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:46.769332    4476 generic.go:356] "Generic (PLEG): container finished" podID="6378f1c0-7559-4bb1-85aa-262a96899475" containerID="2da6f2fd091ab75c358f89260e7069ef12f5c72535203063d4000756d196bccc" exitCode=0
Nov 02 18:17:46.769806 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:46.769397    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd" event={"ID":"6378f1c0-7559-4bb1-85aa-262a96899475","Type":"ContainerDied","Data":"2da6f2fd091ab75c358f89260e7069ef12f5c72535203063d4000756d196bccc"}
Nov 02 18:17:47.049771 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.049692    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"]
Nov 02 18:17:47.053519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.053502    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.123230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.123195    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"]
Nov 02 18:17:47.162709 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.162666    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.162884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.162824    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.162985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.162967    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsfrl\" (UniqueName: \"kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.264338 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.264296    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.264543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.264390    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.264543 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.264484    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsfrl\" (UniqueName: \"kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.265304 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.265269    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.267697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.267667    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.335477 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.335397    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsfrl\" (UniqueName: \"kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl\") pod \"maas-api-revoke-sub-ec263b49-6ljk7\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.367848 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.367812    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:47.393893 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.393866    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:47.746047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.746014    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"]
Nov 02 18:17:47.747725 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:17:47.747697    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4385c942_66a9_4242_be1b_04623d65699d.slice/crio-807cb4f29b5feb811e0e706ca3d3146c2989d2285969f8ddc1ff4ccc77ff0d81 WatchSource:0}: Error finding container 807cb4f29b5feb811e0e706ca3d3146c2989d2285969f8ddc1ff4ccc77ff0d81: Status 404 returned error can't find the container with id 807cb4f29b5feb811e0e706ca3d3146c2989d2285969f8ddc1ff4ccc77ff0d81
Nov 02 18:17:47.778090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.778058    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7" event={"ID":"4385c942-66a9-4242-be1b-04623d65699d","Type":"ContainerStarted","Data":"807cb4f29b5feb811e0e706ca3d3146c2989d2285969f8ddc1ff4ccc77ff0d81"}
Nov 02 18:17:47.960208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:47.960185    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:48.073317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.073286    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca\") pod \"6378f1c0-7559-4bb1-85aa-262a96899475\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") "
Nov 02 18:17:48.073508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.073489    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs\" (UniqueName: \"kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs\") pod \"6378f1c0-7559-4bb1-85aa-262a96899475\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") "
Nov 02 18:17:48.073578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.073551    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca\") pod \"6378f1c0-7559-4bb1-85aa-262a96899475\" (UID: \"6378f1c0-7559-4bb1-85aa-262a96899475\") "
Nov 02 18:17:48.075220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.075192    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca" pod "6378f1c0-7559-4bb1-85aa-262a96899475" (UID: "6378f1c0-7559-4bb1-85aa-262a96899475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:17:48.078583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.078558    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca" pod "6378f1c0-7559-4bb1-85aa-262a96899475" (UID: "6378f1c0-7559-4bb1-85aa-262a96899475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:48.078940 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.078906    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs" pod "6378f1c0-7559-4bb1-85aa-262a96899475" (UID: "6378f1c0-7559-4bb1-85aa-262a96899475"). InnerVolumeSpecName "kube-api-access-wrjxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:48.175500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.175467    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjxs\" (UniqueName: \"kubernetes.io/projected/6378f1c0-7559-4bb1-85aa-262a96899475-kube-api-access-wrjxs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:48.175500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.175499    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:48.175500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.175513    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6378f1c0-7559-4bb1-85aa-262a96899475-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:48.787581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.787553    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"
Nov 02 18:17:48.788046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.787588    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd" event={"ID":"6378f1c0-7559-4bb1-85aa-262a96899475","Type":"ContainerDied","Data":"ee19db0dd86d77eb76f2af18c0377152209d79c08aa9d142c5e222225e1d6f4f"}
Nov 02 18:17:48.788046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.787653    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee19db0dd86d77eb76f2af18c0377152209d79c08aa9d142c5e222225e1d6f4f"
Nov 02 18:17:48.789507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.789483    4476 generic.go:356] "Generic (PLEG): container finished" podID="4385c942-66a9-4242-be1b-04623d65699d" containerID="b7bbf90ad2a3f2483091099770066fa45a3c5417a6522724481e99a9cc06d2bd" exitCode=0
Nov 02 18:17:48.789694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:48.789571    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7" event={"ID":"4385c942-66a9-4242-be1b-04623d65699d","Type":"ContainerDied","Data":"b7bbf90ad2a3f2483091099770066fa45a3c5417a6522724481e99a9cc06d2bd"}
Nov 02 18:17:49.636619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.636583    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:49.936033 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.935962    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:17:49.942536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.942513    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:17:49.973448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.973422    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:49.999533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.999507    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca\") pod \"4385c942-66a9-4242-be1b-04623d65699d\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") "
Nov 02 18:17:49.999714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.999569    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca\") pod \"4385c942-66a9-4242-be1b-04623d65699d\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") "
Nov 02 18:17:49.999714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.999639    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl\" (UniqueName: \"kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl\") pod \"4385c942-66a9-4242-be1b-04623d65699d\" (UID: \"4385c942-66a9-4242-be1b-04623d65699d\") "
Nov 02 18:17:50.000005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:49.999977    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca" pod "4385c942-66a9-4242-be1b-04623d65699d" (UID: "4385c942-66a9-4242-be1b-04623d65699d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:17:50.000122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.000109    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4385c942-66a9-4242-be1b-04623d65699d-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:50.001862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.001842    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca" pod "4385c942-66a9-4242-be1b-04623d65699d" (UID: "4385c942-66a9-4242-be1b-04623d65699d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:50.001862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.001850    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl" pod "4385c942-66a9-4242-be1b-04623d65699d" (UID: "4385c942-66a9-4242-be1b-04623d65699d"). InnerVolumeSpecName "kube-api-access-bsfrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:50.101591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.101553    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4385c942-66a9-4242-be1b-04623d65699d-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:50.101591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.101579    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsfrl\" (UniqueName: \"kubernetes.io/projected/4385c942-66a9-4242-be1b-04623d65699d-kube-api-access-bsfrl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:50.804098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.804071    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"
Nov 02 18:17:50.804252 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.804105    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7" event={"ID":"4385c942-66a9-4242-be1b-04623d65699d","Type":"ContainerDied","Data":"807cb4f29b5feb811e0e706ca3d3146c2989d2285969f8ddc1ff4ccc77ff0d81"}
Nov 02 18:17:50.804252 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:50.804161    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807cb4f29b5feb811e0e706ca3d3146c2989d2285969f8ddc1ff4ccc77ff0d81"
Nov 02 18:17:51.095224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095147    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-11f450"]
Nov 02 18:17:51.095693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095671    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4385c942-66a9-4242-be1b-04623d65699d" containerName="revoke-keys"
Nov 02 18:17:51.095693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095688    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4385c942-66a9-4242-be1b-04623d65699d" containerName="revoke-keys"
Nov 02 18:17:51.095861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095701    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6378f1c0-7559-4bb1-85aa-262a96899475" containerName="revoke-keys"
Nov 02 18:17:51.095861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095706    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6378f1c0-7559-4bb1-85aa-262a96899475" containerName="revoke-keys"
Nov 02 18:17:51.095861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095787    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4385c942-66a9-4242-be1b-04623d65699d" containerName="revoke-keys"
Nov 02 18:17:51.095861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.095797    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6378f1c0-7559-4bb1-85aa-262a96899475" containerName="revoke-keys"
Nov 02 18:17:51.100569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.100551    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:51.141863 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.141832    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:51.157359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.157314    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4gwc\""
Nov 02 18:17:51.168768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.168734    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-11f450"]
Nov 02 18:17:51.212839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.212803    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrp4\" (UniqueName: \"kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4\") pod \"test-curl-7520-11f450\" (UID: \"1243b688-7604-4d0e-aad1-bb11715c023d\") " pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:51.313605 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.313571    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrp4\" (UniqueName: \"kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4\") pod \"test-curl-7520-11f450\" (UID: \"1243b688-7604-4d0e-aad1-bb11715c023d\") " pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:51.377664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.371489    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrp4\" (UniqueName: \"kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4\") pod \"test-curl-7520-11f450\" (UID: \"1243b688-7604-4d0e-aad1-bb11715c023d\") " pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:51.413554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.413523    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:51.840288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.840264    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-11f450"]
Nov 02 18:17:51.843330 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:17:51.843297    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1243b688_7604_4d0e_aad1_bb11715c023d.slice/crio-c9731fa88b0f0391efc2a36f3a99f957dccfad33751cb6ee4be27232b1f338b1 WatchSource:0}: Error finding container c9731fa88b0f0391efc2a36f3a99f957dccfad33751cb6ee4be27232b1f338b1: Status 404 returned error can't find the container with id c9731fa88b0f0391efc2a36f3a99f957dccfad33751cb6ee4be27232b1f338b1
Nov 02 18:17:51.845931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.845915    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:17:51.901480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:51.901399    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:52.082308 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:52.082260    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:52.820549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:52.820518    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-11f450" event={"ID":"1243b688-7604-4d0e-aad1-bb11715c023d","Type":"ContainerStarted","Data":"c9731fa88b0f0391efc2a36f3a99f957dccfad33751cb6ee4be27232b1f338b1"}
Nov 02 18:17:53.828419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:53.828382    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-11f450" event={"ID":"1243b688-7604-4d0e-aad1-bb11715c023d","Type":"ContainerStarted","Data":"843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905"}
Nov 02 18:17:53.941582 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:53.941529    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7520-11f450" podStartSLOduration=1.782750907 podStartE2EDuration="2.941516277s" totalImagesPullingTime="1.15876537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:53.938860067 +0000 UTC m=+3304.779466003" watchObservedRunningTime="2026-11-02 18:17:53.941516277 +0000 UTC m=+3304.782122212"
Nov 02 18:17:54.875322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.875290    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"]
Nov 02 18:17:54.879764 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.879745    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:54.932259 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.932227    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:17:54.954027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.953986    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:54.954027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.954024    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:54.954283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.954147    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2svp\" (UniqueName: \"kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:54.981532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.981502    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:17:54.988007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:54.987966    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"]
Nov 02 18:17:55.055029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.054994    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2svp\" (UniqueName: \"kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.055196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.055146    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.055196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.055174    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.055970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.055942    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.057758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.057732    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.111356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.111317    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2svp\" (UniqueName: \"kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp\") pod \"maas-api-revoke-sub-495c8ff8-p6bhl\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.193984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.193903    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:55.352104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.352066    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-11f450"]
Nov 02 18:17:55.356368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.356334    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-11f450"]
Nov 02 18:17:55.497433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.497362    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-e34c2c"]
Nov 02 18:17:55.497906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.497890    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" containerName="test-curl-7520-11f450"
Nov 02 18:17:55.497967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.497908    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" containerName="test-curl-7520-11f450"
Nov 02 18:17:55.498050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.498040    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" containerName="test-curl-7520-11f450"
Nov 02 18:17:55.502468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.502451    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:55.555985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.555952    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-e34c2c"]
Nov 02 18:17:55.560506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.560473    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspnh\" (UniqueName: \"kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh\") pod \"test-curl-7520-e34c2c\" (UID: \"8d541c87-d43e-41fc-97a2-ae3f153565df\") " pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:55.574846 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:17:55.574821    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3400ce_7a44_4e1e_84c1_7f0f732dc273.slice/crio-f9114c4d9a2396f2d82eb1e5199c642fedeacbb702a70337c4712246b6d9c851 WatchSource:0}: Error finding container f9114c4d9a2396f2d82eb1e5199c642fedeacbb702a70337c4712246b6d9c851: Status 404 returned error can't find the container with id f9114c4d9a2396f2d82eb1e5199c642fedeacbb702a70337c4712246b6d9c851
Nov 02 18:17:55.578663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.575719    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"]
Nov 02 18:17:55.662189 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.662155    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspnh\" (UniqueName: \"kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh\") pod \"test-curl-7520-e34c2c\" (UID: \"8d541c87-d43e-41fc-97a2-ae3f153565df\") " pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:55.718751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.718723    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspnh\" (UniqueName: \"kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh\") pod \"test-curl-7520-e34c2c\" (UID: \"8d541c87-d43e-41fc-97a2-ae3f153565df\") " pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:55.815946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.815891    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:55.875380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.875342    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl" event={"ID":"bc3400ce-7a44-4e1e-84c1-7f0f732dc273","Type":"ContainerStarted","Data":"f9114c4d9a2396f2d82eb1e5199c642fedeacbb702a70337c4712246b6d9c851"}
Nov 02 18:17:55.875886 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:55.875419    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7520-11f450" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" containerName="test-curl-7520-11f450" containerID="cri-o://843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905" gracePeriod=2
Nov 02 18:17:56.255113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.255070    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-e34c2c"]
Nov 02 18:17:56.256513 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:17:56.256482    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d541c87_d43e_41fc_97a2_ae3f153565df.slice/crio-5169af7b503d6eb2ef8aa2e6258737b356e9528760a52db3c08330348b3a06fc WatchSource:0}: Error finding container 5169af7b503d6eb2ef8aa2e6258737b356e9528760a52db3c08330348b3a06fc: Status 404 returned error can't find the container with id 5169af7b503d6eb2ef8aa2e6258737b356e9528760a52db3c08330348b3a06fc
Nov 02 18:17:56.694140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.694104    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:56.885081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.885049    4476 generic.go:356] "Generic (PLEG): container finished" podID="bc3400ce-7a44-4e1e-84c1-7f0f732dc273" containerID="67c57a7bcf54dee122a2a829dbeb7fb0b70e89605f400ded0217c60e69ec60c4" exitCode=0
Nov 02 18:17:56.885581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.885131    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl" event={"ID":"bc3400ce-7a44-4e1e-84c1-7f0f732dc273","Type":"ContainerDied","Data":"67c57a7bcf54dee122a2a829dbeb7fb0b70e89605f400ded0217c60e69ec60c4"}
Nov 02 18:17:56.886858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.886832    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-e34c2c" event={"ID":"8d541c87-d43e-41fc-97a2-ae3f153565df","Type":"ContainerStarted","Data":"0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d"}
Nov 02 18:17:56.886981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.886871    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-e34c2c" event={"ID":"8d541c87-d43e-41fc-97a2-ae3f153565df","Type":"ContainerStarted","Data":"5169af7b503d6eb2ef8aa2e6258737b356e9528760a52db3c08330348b3a06fc"}
Nov 02 18:17:56.968833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:56.968790    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" podUID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" containerName="main" containerID="cri-o://35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3" gracePeriod=45
Nov 02 18:17:57.108305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.108252    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7520-e34c2c" podStartSLOduration=2.108234109 podStartE2EDuration="2.108234109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:57.106837332 +0000 UTC m=+3307.947443266" watchObservedRunningTime="2026-11-02 18:17:57.108234109 +0000 UTC m=+3307.948840044"
Nov 02 18:17:57.243948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.243918    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:17:57.285287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.285257    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home\") pod \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") "
Nov 02 18:17:57.286028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.285858    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr\" (UniqueName: \"kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr\") pod \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") "
Nov 02 18:17:57.286028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.285957    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home" pod "07fc693e-b0b5-4be4-9fad-d207ab00f3c4" (UID: "07fc693e-b0b5-4be4-9fad-d207ab00f3c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:57.286028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.286003    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs\") pod \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") "
Nov 02 18:17:57.286299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.286049    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir\") pod \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") "
Nov 02 18:17:57.286299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.286217    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache\") pod \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") "
Nov 02 18:17:57.286299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.286249    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm\") pod \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\" (UID: \"07fc693e-b0b5-4be4-9fad-d207ab00f3c4\") "
Nov 02 18:17:57.286999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.286702    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir" pod "07fc693e-b0b5-4be4-9fad-d207ab00f3c4" (UID: "07fc693e-b0b5-4be4-9fad-d207ab00f3c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:57.287092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.287070    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:57.287149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.287092    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:57.287741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.287704    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache" pod "07fc693e-b0b5-4be4-9fad-d207ab00f3c4" (UID: "07fc693e-b0b5-4be4-9fad-d207ab00f3c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:57.298507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.298467    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs" pod "07fc693e-b0b5-4be4-9fad-d207ab00f3c4" (UID: "07fc693e-b0b5-4be4-9fad-d207ab00f3c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:57.300274 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.300247    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr" pod "07fc693e-b0b5-4be4-9fad-d207ab00f3c4" (UID: "07fc693e-b0b5-4be4-9fad-d207ab00f3c4"). InnerVolumeSpecName "kube-api-access-p4cnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:57.300387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.300368    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm" pod "07fc693e-b0b5-4be4-9fad-d207ab00f3c4" (UID: "07fc693e-b0b5-4be4-9fad-d207ab00f3c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:57.387862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.387824    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:57.387862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.387849    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:57.387862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.387864    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:57.388661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.387879    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4cnr\" (UniqueName: \"kubernetes.io/projected/07fc693e-b0b5-4be4-9fad-d207ab00f3c4-kube-api-access-p4cnr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:57.896405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.896377    4476 generic.go:356] "Generic (PLEG): container finished" podID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" containerID="35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3" exitCode=0
Nov 02 18:17:57.896778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.896432    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"
Nov 02 18:17:57.896778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.896454    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" event={"ID":"07fc693e-b0b5-4be4-9fad-d207ab00f3c4","Type":"ContainerDied","Data":"35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3"}
Nov 02 18:17:57.896778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.896500    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" event={"ID":"07fc693e-b0b5-4be4-9fad-d207ab00f3c4","Type":"ContainerDied","Data":"9704052e96fade9eb9acd495106272271822eb8873072aa4830ca02acce87422"}
Nov 02 18:17:57.896778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.896522    4476 scope.go:122] "RemoveContainer" containerID="35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3"
Nov 02 18:17:57.941571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.941547    4476 scope.go:122] "RemoveContainer" containerID="35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3"
Nov 02 18:17:57.941958 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:57.941935    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3\": container with ID starting with 35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3 not found: ID does not exist" containerID="35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3"
Nov 02 18:17:57.942069 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:57.941966    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3"} err="failed to get container status \"35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3\": rpc error: code = NotFound desc = could not find container \"35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3\": container with ID starting with 35e95425d9496e8b63c6ac1ff00eb661d8dc605c2498f2436fdeb5e57ec150d3 not found: ID does not exist"
Nov 02 18:17:58.026787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.026765    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:58.061620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.061593    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"]
Nov 02 18:17:58.095712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.095686    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca\") pod \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") "
Nov 02 18:17:58.095863 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.095778    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca\") pod \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") "
Nov 02 18:17:58.095998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.095980    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp\" (UniqueName: \"kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp\") pod \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\" (UID: \"bc3400ce-7a44-4e1e-84c1-7f0f732dc273\") "
Nov 02 18:17:58.096106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.096087    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca" pod "bc3400ce-7a44-4e1e-84c1-7f0f732dc273" (UID: "bc3400ce-7a44-4e1e-84c1-7f0f732dc273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:17:58.096423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.096361    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.097977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.097943    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca" pod "bc3400ce-7a44-4e1e-84c1-7f0f732dc273" (UID: "bc3400ce-7a44-4e1e-84c1-7f0f732dc273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:58.098075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.098055    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp" pod "bc3400ce-7a44-4e1e-84c1-7f0f732dc273" (UID: "bc3400ce-7a44-4e1e-84c1-7f0f732dc273"). InnerVolumeSpecName "kube-api-access-h2svp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:58.109169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.109147    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r"]
Nov 02 18:17:58.114255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.114239    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:58.207652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.200351    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4\" (UniqueName: \"kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4\") pod \"1243b688-7604-4d0e-aad1-bb11715c023d\" (UID: \"1243b688-7604-4d0e-aad1-bb11715c023d\") "
Nov 02 18:17:58.207652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.200844    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2svp\" (UniqueName: \"kubernetes.io/projected/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-kube-api-access-h2svp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.207652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.200866    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3400ce-7a44-4e1e-84c1-7f0f732dc273-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.208074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.208039    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-dd92ac/auth-test-model-dd92ac-kserve-6b9fcd8d9c-mr88r" podUID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:17:58.210735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.210703    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4" pod "1243b688-7604-4d0e-aad1-bb11715c023d" (UID: "1243b688-7604-4d0e-aad1-bb11715c023d"). InnerVolumeSpecName "kube-api-access-7xrp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:58.301918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.301880    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xrp4\" (UniqueName: \"kubernetes.io/projected/1243b688-7604-4d0e-aad1-bb11715c023d-kube-api-access-7xrp4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:17:58.584749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.584655    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-e34c2c"]
Nov 02 18:17:58.590201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.590174    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-e34c2c"]
Nov 02 18:17:58.663251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.663226    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:58.905643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.905589    4476 generic.go:356] "Generic (PLEG): container finished" podID="1243b688-7604-4d0e-aad1-bb11715c023d" containerID="843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905" exitCode=137
Nov 02 18:17:58.906081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.905658    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:58.906081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.905682    4476 scope.go:122] "RemoveContainer" containerID="843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905"
Nov 02 18:17:58.907696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.907298    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"
Nov 02 18:17:58.907696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.907327    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl" event={"ID":"bc3400ce-7a44-4e1e-84c1-7f0f732dc273","Type":"ContainerDied","Data":"f9114c4d9a2396f2d82eb1e5199c642fedeacbb702a70337c4712246b6d9c851"}
Nov 02 18:17:58.907696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.907382    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9114c4d9a2396f2d82eb1e5199c642fedeacbb702a70337c4712246b6d9c851"
Nov 02 18:17:58.907696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.907503    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7520-e34c2c" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" containerName="test-curl-7520-e34c2c" containerID="cri-o://0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d" gracePeriod=2
Nov 02 18:17:58.916656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.916603    4476 scope.go:122] "RemoveContainer" containerID="843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905"
Nov 02 18:17:58.916977 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:58.916951    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905\": container with ID starting with 843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905 not found: ID does not exist" containerID="843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905"
Nov 02 18:17:58.917091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:58.916981    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905"} err="failed to get container status \"843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905\": rpc error: code = NotFound desc = could not find container \"843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905\": container with ID starting with 843dad0f74ddc24030cc6fb931113427f7340d3fba594c68816bf2493e6eb905 not found: ID does not exist"
Nov 02 18:17:58.958308 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:58.958269    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-e34c2c\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:59.059214 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:59.059174    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:17:59.060226 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:59.060204    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-11f450\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:17:59.110710 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:59.110672    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-e34c2c\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:59.383905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:59.383874    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" path="/var/lib/kubelet/pods/07fc693e-b0b5-4be4-9fad-d207ab00f3c4/volumes"
Nov 02 18:17:59.384284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:17:59.384270    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" path="/var/lib/kubelet/pods/1243b688-7604-4d0e-aad1-bb11715c023d/volumes"
Nov 02 18:17:59.430738 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:59.430706    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-e34c2c\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:17:59.481158 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:17:59.481121    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7520-11f450\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="1243b688-7604-4d0e-aad1-bb11715c023d" pod="opendatahub/test-curl-7520-11f450"
Nov 02 18:18:00.178926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:00.178872    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:00.941699 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:00.941511    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:01.191838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.191812    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:18:01.196268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.196239    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:01.346440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.346401    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh\" (UniqueName: \"kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh\") pod \"8d541c87-d43e-41fc-97a2-ae3f153565df\" (UID: \"8d541c87-d43e-41fc-97a2-ae3f153565df\") "
Nov 02 18:18:01.349432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.349389    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh" pod "8d541c87-d43e-41fc-97a2-ae3f153565df" (UID: "8d541c87-d43e-41fc-97a2-ae3f153565df"). InnerVolumeSpecName "kube-api-access-lspnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:01.384089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.384052    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" path="/var/lib/kubelet/pods/8d541c87-d43e-41fc-97a2-ae3f153565df/volumes"
Nov 02 18:18:01.448756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.448660    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lspnh\" (UniqueName: \"kubernetes.io/projected/8d541c87-d43e-41fc-97a2-ae3f153565df-kube-api-access-lspnh\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:01.515260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.515190    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:01.980118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.980079    4476 generic.go:356] "Generic (PLEG): container finished" podID="8d541c87-d43e-41fc-97a2-ae3f153565df" containerID="0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d" exitCode=137
Nov 02 18:18:01.980364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.980352    4476 scope.go:122] "RemoveContainer" containerID="0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d"
Nov 02 18:18:01.980502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.980490    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-e34c2c"
Nov 02 18:18:01.994354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.994330    4476 scope.go:122] "RemoveContainer" containerID="0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d"
Nov 02 18:18:01.994682 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:01.994660    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d\": container with ID starting with 0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d not found: ID does not exist" containerID="0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d"
Nov 02 18:18:01.994754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:01.994690    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d"} err="failed to get container status \"0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d\": rpc error: code = NotFound desc = could not find container \"0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d\": container with ID starting with 0e503700a7e91fe5ff707f462861cb876a8858207405959b1ee9c4b615d9fa0d not found: ID does not exist"
Nov 02 18:18:02.920669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:02.920610    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:03.831812 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:03.831761    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:04.059035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:04.058985    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:05.668116 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:05.668077    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:05.876152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:05.876120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:06.813395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:06.813345    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:07.224061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:07.224028    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"]
Nov 02 18:18:07.224313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:07.224286    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy" containerID="cri-o://3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516" gracePeriod=30
Nov 02 18:18:08.771029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:08.770988    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:08.960347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:08.960314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:09.157888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:09.157856    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:12.098672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.098617    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:12.481303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.481279    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:18:12.577892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.577854    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.577892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.577893    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.577924    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.577967    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578074    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578106    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578189    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:12.578371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578339    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578359    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:12.578544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578458    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578537    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:12.578701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578562    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket\") pod \"be920c1e-eb2a-4e61-bdee-52150339ef05\" (UID: \"be920c1e-eb2a-4e61-bdee-52150339ef05\") "
Nov 02 18:18:12.578975 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578943    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:12.579120 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.578989    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:18:12.579190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.579140    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.579190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.579164    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.579287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.579200    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be920c1e-eb2a-4e61-bdee-52150339ef05-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.579287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.579216    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.579287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.579232    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.580660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.580638    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:18:12.580753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.580725    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:12.580918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.580902    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:12.580987 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.580942    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt" pod "be920c1e-eb2a-4e61-bdee-52150339ef05" (UID: "be920c1e-eb2a-4e61-bdee-52150339ef05"). InnerVolumeSpecName "kube-api-access-wfjrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:12.680931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.680850    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.680931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.680890    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.680931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.680906    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjrt\" (UniqueName: \"kubernetes.io/projected/be920c1e-eb2a-4e61-bdee-52150339ef05-kube-api-access-wfjrt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:12.680931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:12.680920    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be920c1e-eb2a-4e61-bdee-52150339ef05-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:13.073213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.073123    4476 generic.go:356] "Generic (PLEG): container finished" podID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerID="3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516" exitCode=0
Nov 02 18:18:13.073213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.073168    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" event={"ID":"be920c1e-eb2a-4e61-bdee-52150339ef05","Type":"ContainerDied","Data":"3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516"}
Nov 02 18:18:13.073213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.073173    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"
Nov 02 18:18:13.073213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.073211    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm" event={"ID":"be920c1e-eb2a-4e61-bdee-52150339ef05","Type":"ContainerDied","Data":"dcbcff7d83f79cdf6728f85c7de8eb35eaa3c5c94f7e9bb4395451c32825d5e6"}
Nov 02 18:18:13.073535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.073232    4476 scope.go:122] "RemoveContainer" containerID="3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516"
Nov 02 18:18:13.084086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.084060    4476 scope.go:122] "RemoveContainer" containerID="3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516"
Nov 02 18:18:13.084379 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:13.084355    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516\": container with ID starting with 3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516 not found: ID does not exist" containerID="3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516"
Nov 02 18:18:13.084448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.084388    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516"} err="failed to get container status \"3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516\": rpc error: code = NotFound desc = could not find container \"3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516\": container with ID starting with 3e3fa84e978b3c88ee8850250540405887af5296c5f0edc4c776e99ec3490516 not found: ID does not exist"
Nov 02 18:18:13.238925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.238894    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"]
Nov 02 18:18:13.242145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.242121    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-lrvnm"]
Nov 02 18:18:13.384531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:13.384498    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" path="/var/lib/kubelet/pods/be920c1e-eb2a-4e61-bdee-52150339ef05/volumes"
Nov 02 18:18:16.404658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:16.404591    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"]
Nov 02 18:18:16.405094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:16.404879    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" containerID="cri-o://9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9" gracePeriod=30
Nov 02 18:18:17.012299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:17.012265    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:18.486227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:18.486191    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"]
Nov 02 18:18:18.548453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:18.548416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:21.263608 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.263568    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:18:21.264182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.263711    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:18:21.420921 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.420891    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"]
Nov 02 18:18:21.488782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.488735    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:21.700098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.699968    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:21.712232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.712201    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:18:21.787471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787381    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.787471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787456    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.787731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787502    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.787731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787534    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.787731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787604    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.787731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787680    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.787731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787708    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.788038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787749    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.788038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787793    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs\") pod \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\" (UID: \"2d6b56a5-9cc4-461a-98fa-116c89208ffe\") "
Nov 02 18:18:21.788038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787851    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:21.788038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.787899    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:21.788269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.788069    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:21.788491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.788290    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.788491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.788308    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.788491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.788322    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.788491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.788334    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:18:21.788491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.788425    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:21.789875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.789852    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "kube-api-access-kptf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:21.790066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.790051    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:21.790272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.790255    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:21.790705 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.790674    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo" pod "2d6b56a5-9cc4-461a-98fa-116c89208ffe" (UID: "2d6b56a5-9cc4-461a-98fa-116c89208ffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:18:21.889696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.889607    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.889696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.889694    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptf2\" (UniqueName: \"kubernetes.io/projected/2d6b56a5-9cc4-461a-98fa-116c89208ffe-kube-api-access-kptf2\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.889911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.889712    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.889911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.889725    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.889911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.889738    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d6b56a5-9cc4-461a-98fa-116c89208ffe-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:21.889911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:21.889749    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d6b56a5-9cc4-461a-98fa-116c89208ffe-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:22.136744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.136713    4476 generic.go:356] "Generic (PLEG): container finished" podID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerID="9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9" exitCode=0
Nov 02 18:18:22.136906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.136765    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"
Nov 02 18:18:22.136906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.136802    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" event={"ID":"2d6b56a5-9cc4-461a-98fa-116c89208ffe","Type":"ContainerDied","Data":"9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9"}
Nov 02 18:18:22.136906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.136860    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln" event={"ID":"2d6b56a5-9cc4-461a-98fa-116c89208ffe","Type":"ContainerDied","Data":"7eb98737633414f9085d4bdfd28af579e16c9b9d2903d6f70094ba367b8fdbc3"}
Nov 02 18:18:22.136906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.136884    4476 scope.go:122] "RemoveContainer" containerID="9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9"
Nov 02 18:18:22.147967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.147950    4476 scope.go:122] "RemoveContainer" containerID="9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9"
Nov 02 18:18:22.148258 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:22.148237    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9\": container with ID starting with 9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9 not found: ID does not exist" containerID="9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9"
Nov 02 18:18:22.148308 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.148267    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9"} err="failed to get container status \"9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9\": rpc error: code = NotFound desc = could not find container \"9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9\": container with ID starting with 9f27c9a055b8fb7abb24089a0aa1a114d1838ba3ceeb0640222fd37b174f8dd9 not found: ID does not exist"
Nov 02 18:18:22.300902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.300870    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"]
Nov 02 18:18:22.306866 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.306842    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-bvcln"]
Nov 02 18:18:22.500516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:22.500397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:23.383520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:23.383479    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" path="/var/lib/kubelet/pods/2d6b56a5-9cc4-461a-98fa-116c89208ffe/volumes"
Nov 02 18:18:23.880446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:23.880413    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:24.336529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.336435    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"]
Nov 02 18:18:24.337041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337022    4476 cpu_manager.go:414] "containerMap: removing container" podUID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" containerName="main"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337043    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" containerName="main"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337073    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bc3400ce-7a44-4e1e-84c1-7f0f732dc273" containerName="revoke-keys"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337080    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3400ce-7a44-4e1e-84c1-7f0f732dc273" containerName="revoke-keys"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337096    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" containerName="test-curl-7520-e34c2c"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337103    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" containerName="test-curl-7520-e34c2c"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337119    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy"
Nov 02 18:18:24.337149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337126    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337156    4476 cpu_manager.go:414] "containerMap: removing container" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337165    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337303    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3400ce-7a44-4e1e-84c1-7f0f732dc273" containerName="revoke-keys"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337316    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d541c87-d43e-41fc-97a2-ae3f153565df" containerName="test-curl-7520-e34c2c"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337326    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6b56a5-9cc4-461a-98fa-116c89208ffe" containerName="istio-proxy"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337342    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="be920c1e-eb2a-4e61-bdee-52150339ef05" containerName="istio-proxy"
Nov 02 18:18:24.337509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.337354    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fc693e-b0b5-4be4-9fad-d207ab00f3c4" containerName="main"
Nov 02 18:18:24.342172 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.342151    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418306    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418390    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418500    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418529    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418575    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwdt\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418659    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418728    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.419535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.418780    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.449968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.449932    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"]
Nov 02 18:18:24.480731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.479613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:24.494323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.494267    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a9496c-openshift-default-dockercfg-2xcbj\""
Nov 02 18:18:24.520116 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520079    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520174    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520201    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520239    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520268    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwdt\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520327    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520382    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520418    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.520620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.520489    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.521230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.521202    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.521603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.521574    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.521734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.521600    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.523010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.522891    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.523010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.522925    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.524950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.524924    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.527482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.527459    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.579333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.579305    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwdt\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.579525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.579307    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token\") pod \"e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") " pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.661674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.661605    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:24.793686 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:24.793492    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:25.107041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:25.107012    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"]
Nov 02 18:18:25.107715 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:18:25.107689    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144aff60_b302_4731_9031_da37c7aebea0.slice/crio-1cb5efe323815127ea1b46dee387c1e938a0e81392a76924c60f5089a687bc06 WatchSource:0}: Error finding container 1cb5efe323815127ea1b46dee387c1e938a0e81392a76924c60f5089a687bc06: Status 404 returned error can't find the container with id 1cb5efe323815127ea1b46dee387c1e938a0e81392a76924c60f5089a687bc06
Nov 02 18:18:25.110862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:25.110831    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:18:25.110935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:25.110906    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:18:25.110971 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:25.110933    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:18:25.178181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:25.178154    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" event={"ID":"144aff60-b302-4731-9031-da37c7aebea0","Type":"ContainerStarted","Data":"1cb5efe323815127ea1b46dee387c1e938a0e81392a76924c60f5089a687bc06"}
Nov 02 18:18:26.187318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:26.187280    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" event={"ID":"144aff60-b302-4731-9031-da37c7aebea0","Type":"ContainerStarted","Data":"445b2a58d2a5908721647d1d44f1268cebdc691456bb043d4a72a436185d8c58"}
Nov 02 18:18:26.357482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:26.357420    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podStartSLOduration=2.357400769 podStartE2EDuration="2.357400769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:26.353432246 +0000 UTC m=+3337.194038181" watchObservedRunningTime="2026-11-02 18:18:26.357400769 +0000 UTC m=+3337.198006704"
Nov 02 18:18:26.400680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:26.400643    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:26.662105 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:26.662073    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:26.669285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:26.669258    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:27.194131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:27.194100    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:27.195103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:27.195084    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:18:29.060604 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:29.060561    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:18:29.108216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:29.108186    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:29.301766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:29.301737    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:31.142317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:31.142273    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:31.530134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:31.530048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:32.518929 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:32.518888    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:33.259475 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.259440    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"]
Nov 02 18:18:33.264855 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.264825    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.349027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.348986    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cd738b-openshift-default-dockercfg-lbl8s\""
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.443781    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444298    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444341    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444380    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444421    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444463    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444504    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444587    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444700    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.448679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.444756    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.459102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.459069    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"]
Nov 02 18:18:33.521029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.519820    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" podUID="9450795a-a2b5-4887-bc85-6315b7aba5d0" containerName="main" containerID="cri-o://aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa" gracePeriod=45
Nov 02 18:18:33.547400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.547365    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.547774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.547747    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.550859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.550834    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.551807 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.551212    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.551983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.551965    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.554014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.552698    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.554014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.553241    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.554014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.553522    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.554014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.547819    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.554014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.548094    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.561550 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.561486    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.562082 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.562055    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.562473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.562456    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.563094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.562382    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.563474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.563452    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.570395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.569406    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.610680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.610648    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.623254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.623223    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token\") pod \"e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") " pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.718117 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.718040    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:33.881592 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.881335    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:33.890938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:33.890901    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:18:34.070681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.070613    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm\") pod \"9450795a-a2b5-4887-bc85-6315b7aba5d0\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") "
Nov 02 18:18:34.070874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.070808    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir\") pod \"9450795a-a2b5-4887-bc85-6315b7aba5d0\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") "
Nov 02 18:18:34.070874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.070852    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs\") pod \"9450795a-a2b5-4887-bc85-6315b7aba5d0\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") "
Nov 02 18:18:34.070874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.070896    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home\") pod \"9450795a-a2b5-4887-bc85-6315b7aba5d0\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") "
Nov 02 18:18:34.071132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.070938    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache\") pod \"9450795a-a2b5-4887-bc85-6315b7aba5d0\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") "
Nov 02 18:18:34.071132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.071050    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n\" (UniqueName: \"kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n\") pod \"9450795a-a2b5-4887-bc85-6315b7aba5d0\" (UID: \"9450795a-a2b5-4887-bc85-6315b7aba5d0\") "
Nov 02 18:18:34.071132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.071106    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir" pod "9450795a-a2b5-4887-bc85-6315b7aba5d0" (UID: "9450795a-a2b5-4887-bc85-6315b7aba5d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:34.071567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.071321    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home" pod "9450795a-a2b5-4887-bc85-6315b7aba5d0" (UID: "9450795a-a2b5-4887-bc85-6315b7aba5d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:34.071567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.071510    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache" pod "9450795a-a2b5-4887-bc85-6315b7aba5d0" (UID: "9450795a-a2b5-4887-bc85-6315b7aba5d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:34.071567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.071513    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:34.071567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.071548    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:34.074032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.073951    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs" pod "9450795a-a2b5-4887-bc85-6315b7aba5d0" (UID: "9450795a-a2b5-4887-bc85-6315b7aba5d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:34.074032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.073956    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm" pod "9450795a-a2b5-4887-bc85-6315b7aba5d0" (UID: "9450795a-a2b5-4887-bc85-6315b7aba5d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:34.074209 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.074083    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n" pod "9450795a-a2b5-4887-bc85-6315b7aba5d0" (UID: "9450795a-a2b5-4887-bc85-6315b7aba5d0"). InnerVolumeSpecName "kube-api-access-kj46n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:34.172549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.172508    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj46n\" (UniqueName: \"kubernetes.io/projected/9450795a-a2b5-4887-bc85-6315b7aba5d0-kube-api-access-kj46n\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:34.172549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.172550    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:34.172783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.172565    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9450795a-a2b5-4887-bc85-6315b7aba5d0-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:34.172783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.172579    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9450795a-a2b5-4887-bc85-6315b7aba5d0-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:34.250551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.249303    4476 generic.go:356] "Generic (PLEG): container finished" podID="9450795a-a2b5-4887-bc85-6315b7aba5d0" containerID="aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa" exitCode=0
Nov 02 18:18:34.250551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.249364    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"
Nov 02 18:18:34.250551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.249356    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" event={"ID":"9450795a-a2b5-4887-bc85-6315b7aba5d0","Type":"ContainerDied","Data":"aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa"}
Nov 02 18:18:34.250551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.249568    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" event={"ID":"9450795a-a2b5-4887-bc85-6315b7aba5d0","Type":"ContainerDied","Data":"1ffb726bd76d1aaddb06ddd37696100495aaa5e90fcc49885bb2351e1d372fbf"}
Nov 02 18:18:34.250551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.249621    4476 scope.go:122] "RemoveContainer" containerID="aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa"
Nov 02 18:18:34.263421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.263401    4476 scope.go:122] "RemoveContainer" containerID="aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa"
Nov 02 18:18:34.263793 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:34.263769    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa\": container with ID starting with aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa not found: ID does not exist" containerID="aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa"
Nov 02 18:18:34.263899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.263803    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa"} err="failed to get container status \"aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa\": rpc error: code = NotFound desc = could not find container \"aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa\": container with ID starting with aef8c7fa87721f006a94f5d8f309d4b962ea0d9f1c235f40824d0477fa30b1aa not found: ID does not exist"
Nov 02 18:18:34.345533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.345494    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"]
Nov 02 18:18:34.346690 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:18:34.346641    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263b056f_8821_4407_b9cc_b4201d0a9952.slice/crio-e1a22c8cfe5250afc8759365dff31ae7cffcf5f32b7dc485018ec0418069024c WatchSource:0}: Error finding container e1a22c8cfe5250afc8759365dff31ae7cffcf5f32b7dc485018ec0418069024c: Status 404 returned error can't find the container with id e1a22c8cfe5250afc8759365dff31ae7cffcf5f32b7dc485018ec0418069024c
Nov 02 18:18:34.349023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.348990    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:18:34.349119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.349073    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:18:34.349119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.349114    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:18:34.416141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.416114    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"]
Nov 02 18:18:34.423335 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.421830    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv"]
Nov 02 18:18:34.882006 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:34.881964    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-580a21/auth-test-model-580a21-kserve-9c95f57d5-kr9rv" podUID="9450795a-a2b5-4887-bc85-6315b7aba5d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:18:35.259761 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:35.259667    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" event={"ID":"263b056f-8821-4407-b9cc-b4201d0a9952","Type":"ContainerStarted","Data":"2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"}
Nov 02 18:18:35.259761 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:35.259731    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" event={"ID":"263b056f-8821-4407-b9cc-b4201d0a9952","Type":"ContainerStarted","Data":"e1a22c8cfe5250afc8759365dff31ae7cffcf5f32b7dc485018ec0418069024c"}
Nov 02 18:18:35.378272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:35.378209    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podStartSLOduration=2.378195502 podStartE2EDuration="2.378195502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:35.374492565 +0000 UTC m=+3346.215098500" watchObservedRunningTime="2026-11-02 18:18:35.378195502 +0000 UTC m=+3346.218801436"
Nov 02 18:18:35.383176 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:35.383149    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9450795a-a2b5-4887-bc85-6315b7aba5d0" path="/var/lib/kubelet/pods/9450795a-a2b5-4887-bc85-6315b7aba5d0/volumes"
Nov 02 18:18:35.882758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:35.882727    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:35.889079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:35.889058    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:36.268559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.268481    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:36.269391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.269373    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:18:36.446702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.446660    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg" podUID="13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" containerName="main" containerID="cri-o://86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708" gracePeriod=45
Nov 02 18:18:36.560249 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.560176    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:36.744070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.744041    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:36.767300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.767274    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.803542    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir\") pod \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") "
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.803591    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm\") pod \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") "
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.803651    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache\") pod \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") "
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.803719    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home\") pod \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") "
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.803866    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46\" (UniqueName: \"kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46\") pod \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") "
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.803906    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs\") pod \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\" (UID: \"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc\") "
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.804777    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache" pod "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" (UID: "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.804988    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home" pod "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" (UID: "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:36.809967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.805399    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir" pod "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" (UID: "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:36.810562 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.810380    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46" pod "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" (UID: "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc"). InnerVolumeSpecName "kube-api-access-r4m46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:36.812504 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.812466    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs" pod "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" (UID: "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:36.812849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.812826    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm" pod "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" (UID: "13b0849b-2c6e-4601-8f5d-1d715d5f1dbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:18:36.905217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.905178    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4m46\" (UniqueName: \"kubernetes.io/projected/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-kube-api-access-r4m46\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:36.905217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.905210    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:36.905217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.905219    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:36.905698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.905228    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:36.905698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.905237    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:36.905698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.905247    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:36.919711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.919677    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"]
Nov 02 18:18:36.920227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.920209    4476 cpu_manager.go:414] "containerMap: removing container" podUID="13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" containerName="main"
Nov 02 18:18:36.920227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.920226    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" containerName="main"
Nov 02 18:18:36.920389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.920246    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9450795a-a2b5-4887-bc85-6315b7aba5d0" containerName="main"
Nov 02 18:18:36.920389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.920252    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9450795a-a2b5-4887-bc85-6315b7aba5d0" containerName="main"
Nov 02 18:18:36.920389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.920337    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" containerName="main"
Nov 02 18:18:36.920389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.920350    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9450795a-a2b5-4887-bc85-6315b7aba5d0" containerName="main"
Nov 02 18:18:36.923814 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.923797    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:36.984020 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.983979    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a9496c\""
Nov 02 18:18:36.984303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.984280    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a9496c-dockercfg-g9w27\""
Nov 02 18:18:36.985668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:36.985620    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"]
Nov 02 18:18:37.006721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.006667    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.006918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.006817    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7zd\" (UniqueName: \"kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd\") pod \"payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.071711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.071617    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"]
Nov 02 18:18:37.076015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.075988    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.108557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.108514    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.108557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.108568    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpzb\" (UniqueName: \"kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb\") pod \"payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.108824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.108612    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.108888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.108829    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7zd\" (UniqueName: \"kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd\") pod \"payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.109360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.109338    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.132986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.132954    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"]
Nov 02 18:18:37.165914 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.165789    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7zd\" (UniqueName: \"kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd\") pod \"payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.210411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.210369    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpzb\" (UniqueName: \"kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb\") pod \"payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.210638 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.210438    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.211414 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.211391    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.239832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.239801    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:37.270796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.270764    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpzb\" (UniqueName: \"kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb\") pod \"payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.279035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.278995    4476 generic.go:356] "Generic (PLEG): container finished" podID="13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" containerID="86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708" exitCode=0
Nov 02 18:18:37.279903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.279873    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"
Nov 02 18:18:37.280748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.280719    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg" event={"ID":"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc","Type":"ContainerDied","Data":"86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708"}
Nov 02 18:18:37.280916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.280773    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg" event={"ID":"13b0849b-2c6e-4601-8f5d-1d715d5f1dbc","Type":"ContainerDied","Data":"c64f768fe7ba4e3cb6f40b52fcc77a22203b136d2d871ad6ca26dbc7fc567879"}
Nov 02 18:18:37.280916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.280797    4476 scope.go:122] "RemoveContainer" containerID="86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708"
Nov 02 18:18:37.293470 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.293450    4476 scope.go:122] "RemoveContainer" containerID="86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708"
Nov 02 18:18:37.293908 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:37.293873    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708\": container with ID starting with 86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708 not found: ID does not exist" containerID="86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708"
Nov 02 18:18:37.294007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.293923    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708"} err="failed to get container status \"86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708\": rpc error: code = NotFound desc = could not find container \"86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708\": container with ID starting with 86ae5c18abe808588d2b46f48a807fd476da94ee16f6b44b8734b8e8baf91708 not found: ID does not exist"
Nov 02 18:18:37.390890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.390858    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:37.447271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.447241    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"]
Nov 02 18:18:37.453531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.453462    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f7aa2e/auth-test-model-f7aa2e-kserve-5d6b49585-jvptg"]
Nov 02 18:18:37.631354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.631308    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"]
Nov 02 18:18:37.632385 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:18:37.632356    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cd8953_99d2_44d4_9a0c_e4cf14e1d413.slice/crio-5b9b43c6c2c072dff3354237103e2d2c2be53e758c95d95a4c35d1c68c7634f8 WatchSource:0}: Error finding container 5b9b43c6c2c072dff3354237103e2d2c2be53e758c95d95a4c35d1c68c7634f8: Status 404 returned error can't find the container with id 5b9b43c6c2c072dff3354237103e2d2c2be53e758c95d95a4c35d1c68c7634f8
Nov 02 18:18:37.771279 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:37.771249    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"]
Nov 02 18:18:37.773036 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:18:37.773003    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc997153_2a84_43eb_9d52_3c2f00205680.slice/crio-f55ceda009de56ddcb1ac949c9d4c28baa99f751df5e6b6ff015795aa4de2796 WatchSource:0}: Error finding container f55ceda009de56ddcb1ac949c9d4c28baa99f751df5e6b6ff015795aa4de2796: Status 404 returned error can't find the container with id f55ceda009de56ddcb1ac949c9d4c28baa99f751df5e6b6ff015795aa4de2796
Nov 02 18:18:38.288881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.288844    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh" event={"ID":"bc997153-2a84-43eb-9d52-3c2f00205680","Type":"ContainerStarted","Data":"08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37"}
Nov 02 18:18:38.289426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.288921    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh" event={"ID":"bc997153-2a84-43eb-9d52-3c2f00205680","Type":"ContainerStarted","Data":"f55ceda009de56ddcb1ac949c9d4c28baa99f751df5e6b6ff015795aa4de2796"}
Nov 02 18:18:38.289426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.289123    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:38.290933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.290904    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v" event={"ID":"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413","Type":"ContainerStarted","Data":"1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a"}
Nov 02 18:18:38.291069 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.290948    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v" event={"ID":"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413","Type":"ContainerStarted","Data":"5b9b43c6c2c072dff3354237103e2d2c2be53e758c95d95a4c35d1c68c7634f8"}
Nov 02 18:18:38.291130 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.291119    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:38.402935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.402884    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh" podStartSLOduration=1.402870335 podStartE2EDuration="1.402870335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:38.400613143 +0000 UTC m=+3349.241219077" watchObservedRunningTime="2026-11-02 18:18:38.402870335 +0000 UTC m=+3349.243476269"
Nov 02 18:18:38.440530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.440498    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:38.527311 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.527257    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v" podStartSLOduration=2.527244489 podStartE2EDuration="2.527244489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:38.52311821 +0000 UTC m=+3349.363724145" watchObservedRunningTime="2026-11-02 18:18:38.527244489 +0000 UTC m=+3349.367850422"
Nov 02 18:18:38.718113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.718064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:38.920684 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:38.920620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:39.384233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:39.384198    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b0849b-2c6e-4601-8f5d-1d715d5f1dbc" path="/var/lib/kubelet/pods/13b0849b-2c6e-4601-8f5d-1d715d5f1dbc/volumes"
Nov 02 18:18:41.652722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:41.652690    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:41.845317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:41.845278    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:42.028913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.028828    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"]
Nov 02 18:18:42.033132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.033114    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.038155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.038128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:42.073176 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.073144    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2xm\" (UniqueName: \"kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm\") pod \"payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.073369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.073302    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.097531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.097487    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"]
Nov 02 18:18:42.137389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.134509    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cd738b\""
Nov 02 18:18:42.137389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.134722    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cd738b-dockercfg-465k9\""
Nov 02 18:18:42.174549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.174403    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.174801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.174690    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2xm\" (UniqueName: \"kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm\") pod \"payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.175611 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.175439    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.230947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.230918    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2xm\" (UniqueName: \"kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm\") pod \"payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.349266 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.349171    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:42.732545 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:42.732519    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"]
Nov 02 18:18:42.733654 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:18:42.733601    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efe75f2_184b_4ff5_bbf2_7791c96d9417.slice/crio-dc7058b995259057a66e2931882cb343c220c4046b77992c0e1865e766c3be74 WatchSource:0}: Error finding container dc7058b995259057a66e2931882cb343c220c4046b77992c0e1865e766c3be74: Status 404 returned error can't find the container with id dc7058b995259057a66e2931882cb343c220c4046b77992c0e1865e766c3be74
Nov 02 18:18:43.326462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:43.326420    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn" event={"ID":"0efe75f2-184b-4ff5-bbf2-7791c96d9417","Type":"ContainerStarted","Data":"15c3eaf70b514a497ac0f524db58ccb9930a010d1e18ac50016c9b4f8dd4f88d"}
Nov 02 18:18:43.326689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:43.326485    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn" event={"ID":"0efe75f2-184b-4ff5-bbf2-7791c96d9417","Type":"ContainerStarted","Data":"dc7058b995259057a66e2931882cb343c220c4046b77992c0e1865e766c3be74"}
Nov 02 18:18:43.326689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:43.326506    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:43.441590 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:43.441528    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn" podStartSLOduration=2.441513906 podStartE2EDuration="2.441513906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:43.438809444 +0000 UTC m=+3354.279415568" watchObservedRunningTime="2026-11-02 18:18:43.441513906 +0000 UTC m=+3354.282119840"
Nov 02 18:18:46.667215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:46.667177    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:49.301449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:49.301367    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:18:49.301860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:49.301470    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:18:51.386906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.386869    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:51.512362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.512329    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"]
Nov 02 18:18:51.516020 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.516004    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.567786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.567746    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:18:51.567786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.567731    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:18:51.591344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.591314    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.591497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.591355    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.591497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.591437    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwppp\" (UniqueName: \"kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.620726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.620694    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"]
Nov 02 18:18:51.692821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.692739    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.692821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.692779    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.692821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.692800    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwppp\" (UniqueName: \"kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.693546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.693520    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.695369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.695334    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.749612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.749578    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwppp\" (UniqueName: \"kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp\") pod \"maas-api-revoke-keys-e2e-explicit-3dc055-59z6p\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:51.829901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:51.829869    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:52.208225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:52.208189    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"]
Nov 02 18:18:52.209491 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:18:52.209462    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117e569a_c83a_4c6f_aa30_05508919cb0e.slice/crio-63428de470091ebeb3db72844d31738b0ff9fdab0326a929b1d1228f5cbc3c1a WatchSource:0}: Error finding container 63428de470091ebeb3db72844d31738b0ff9fdab0326a929b1d1228f5cbc3c1a: Status 404 returned error can't find the container with id 63428de470091ebeb3db72844d31738b0ff9fdab0326a929b1d1228f5cbc3c1a
Nov 02 18:18:52.402906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:52.402874    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p" event={"ID":"117e569a-c83a-4c6f-aa30-05508919cb0e","Type":"ContainerStarted","Data":"63428de470091ebeb3db72844d31738b0ff9fdab0326a929b1d1228f5cbc3c1a"}
Nov 02 18:18:53.411121 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:53.411081    4476 generic.go:356] "Generic (PLEG): container finished" podID="117e569a-c83a-4c6f-aa30-05508919cb0e" containerID="b5de08975499f7871a96155f9cfb002bdcac1c4b0abc2e57b2b4c846d96fd911" exitCode=0
Nov 02 18:18:53.411487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:53.411161    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p" event={"ID":"117e569a-c83a-4c6f-aa30-05508919cb0e","Type":"ContainerDied","Data":"b5de08975499f7871a96155f9cfb002bdcac1c4b0abc2e57b2b4c846d96fd911"}
Nov 02 18:18:54.337190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.337160    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:18:54.598711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.598654    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:54.727373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.727344    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca\") pod \"117e569a-c83a-4c6f-aa30-05508919cb0e\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") "
Nov 02 18:18:54.727541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.727401    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp\" (UniqueName: \"kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp\") pod \"117e569a-c83a-4c6f-aa30-05508919cb0e\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") "
Nov 02 18:18:54.727541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.727468    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca\") pod \"117e569a-c83a-4c6f-aa30-05508919cb0e\" (UID: \"117e569a-c83a-4c6f-aa30-05508919cb0e\") "
Nov 02 18:18:54.727838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.727809    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca" pod "117e569a-c83a-4c6f-aa30-05508919cb0e" (UID: "117e569a-c83a-4c6f-aa30-05508919cb0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:18:54.729840 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.729806    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp" pod "117e569a-c83a-4c6f-aa30-05508919cb0e" (UID: "117e569a-c83a-4c6f-aa30-05508919cb0e"). InnerVolumeSpecName "kube-api-access-wwppp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:54.729949 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.729842    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca" pod "117e569a-c83a-4c6f-aa30-05508919cb0e" (UID: "117e569a-c83a-4c6f-aa30-05508919cb0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:54.828617 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.828579    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:54.828617 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.828613    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/117e569a-c83a-4c6f-aa30-05508919cb0e-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:54.828824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:54.828623    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwppp\" (UniqueName: \"kubernetes.io/projected/117e569a-c83a-4c6f-aa30-05508919cb0e-kube-api-access-wwppp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:18:55.426705 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:55.426674    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"
Nov 02 18:18:55.426861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:55.426674    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p" event={"ID":"117e569a-c83a-4c6f-aa30-05508919cb0e","Type":"ContainerDied","Data":"63428de470091ebeb3db72844d31738b0ff9fdab0326a929b1d1228f5cbc3c1a"}
Nov 02 18:18:55.426861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:55.426806    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63428de470091ebeb3db72844d31738b0ff9fdab0326a929b1d1228f5cbc3c1a"
Nov 02 18:18:56.290944 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:56.290916    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:58.902717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:58.902674    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:59.058299 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:18:59.058259    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:18:59.797795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.797763    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"]
Nov 02 18:18:59.798312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.798299    4476 cpu_manager.go:414] "containerMap: removing container" podUID="117e569a-c83a-4c6f-aa30-05508919cb0e" containerName="revoke-keys"
Nov 02 18:18:59.798379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.798315    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117e569a-c83a-4c6f-aa30-05508919cb0e" containerName="revoke-keys"
Nov 02 18:18:59.798436 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.798427    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="117e569a-c83a-4c6f-aa30-05508919cb0e" containerName="revoke-keys"
Nov 02 18:18:59.802324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.802299    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.855045 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.855014    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f7aa2e-openshift-default-dockercfg-s9k69\""
Nov 02 18:18:59.888818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.888787    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.888840    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.888957    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.889002    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.889079    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.889096    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7kzm\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889245 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.889129    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.889273    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.889321 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.889301    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.909335 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.909301    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"]
Nov 02 18:18:59.990327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990292    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990340    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990362    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990378    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7kzm\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990411    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990489    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990512    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990559    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990583    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.990978 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990903    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.991018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.990970    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.991085 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.991058    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.991152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.991130    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.991193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.991166    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.993579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.993556    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:18:59.993880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:18:59.993862    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:00.047324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.047294    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:00.047555 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.047531    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7kzm\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:00.116320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.116294    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:00.549737 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:19:00.549704    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d48148_8479_4881_b482_092c1a7920ca.slice/crio-390d94f2690a97be4be82178569695a1a3d414dcec73f115f72ec7bfe04c7a9d WatchSource:0}: Error finding container 390d94f2690a97be4be82178569695a1a3d414dcec73f115f72ec7bfe04c7a9d: Status 404 returned error can't find the container with id 390d94f2690a97be4be82178569695a1a3d414dcec73f115f72ec7bfe04c7a9d
Nov 02 18:19:00.549858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.549741    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"]
Nov 02 18:19:00.553269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.553231    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:19:00.553362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.553312    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:19:00.553429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.553368    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:19:00.961399 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:00.961343    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:01.472069 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:01.472029    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" event={"ID":"e4d48148-8479-4881-b482-092c1a7920ca","Type":"ContainerStarted","Data":"0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f"}
Nov 02 18:19:01.472248 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:01.472089    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" event={"ID":"e4d48148-8479-4881-b482-092c1a7920ca","Type":"ContainerStarted","Data":"390d94f2690a97be4be82178569695a1a3d414dcec73f115f72ec7bfe04c7a9d"}
Nov 02 18:19:01.692144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:01.690924    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podStartSLOduration=2.690902994 podStartE2EDuration="2.690902994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:01.63746088 +0000 UTC m=+3372.478066816" watchObservedRunningTime="2026-11-02 18:19:01.690902994 +0000 UTC m=+3372.531508930"
Nov 02 18:19:02.116700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:02.116664    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:02.118394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:02.118366    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 02 18:19:02.118548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:02.118422    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 02 18:19:03.117409 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:03.117368    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 02 18:19:03.117926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:03.117428    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 02 18:19:04.117087 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:04.117047    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 02 18:19:04.117266 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:04.117123    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 02 18:19:05.051230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:05.051139    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:05.117554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:05.117514    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 02 18:19:05.117799 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:05.117590    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 02 18:19:06.117138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:06.117097    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 02 18:19:06.117777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:06.117178    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 02 18:19:06.740990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:06.740938    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:06.921654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:06.921583    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"]
Nov 02 18:19:06.924515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:06.924476    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:06.973196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:06.973164    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3dc055-59z6p"]
Nov 02 18:19:07.122858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:07.122831    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:07.123374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:07.123358    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:07.124280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:07.124250    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:19:07.285048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:07.285013    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:07.394620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:07.394534    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117e569a-c83a-4c6f-aa30-05508919cb0e" path="/var/lib/kubelet/pods/117e569a-c83a-4c6f-aa30-05508919cb0e/volumes"
Nov 02 18:19:07.854348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:07.854264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:08.312425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:08.312110    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:08.497126 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:08.497093    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:09.012217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:09.012182    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:09.205318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:09.205284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:10.480740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:10.480701    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:10.668981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:10.668943    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:10.863384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:10.863355    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:11.876897 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:11.876852    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:16.647715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:16.647659    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:21.307212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:21.307157    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:21.509241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:21.509210    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:26.299073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:26.299006    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:29.059715 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:19:29.059675    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:19:32.901772 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:32.901733    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:33.115230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:33.115193    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:35.523980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:35.523943    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:35.961520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:35.961493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:40.765715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:40.765677    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:45.470451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:45.470419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:50.144803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:50.144766    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:50.343167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:50.343130    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:50.539827 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:50.539752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:55.141896 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:55.141857    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:59.059500 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:19:59.059458    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:19:59.249325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:59.249289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:59.922193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:19:59.922158    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:00.220909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:00.220833    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:00.415082 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:00.415049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:04.895382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:04.895325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:09.689212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:09.689173    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:11.771519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:11.771471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:12.032506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:12.032402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:12.264349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:12.264184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:12.466530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:12.466487    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:13.407047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:13.407018    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:14.397861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:14.389295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:14.583167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:14.583127    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:14.825213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:14.825134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:17.419906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:17.419871    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:19.533323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:19.533297    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:20.071898 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.071861    4476 generic.go:356] "Generic (PLEG): container finished" podID="144aff60-b302-4731-9031-da37c7aebea0" containerID="445b2a58d2a5908721647d1d44f1268cebdc691456bb043d4a72a436185d8c58" exitCode=0
Nov 02 18:20:20.072101 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.071951    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" event={"ID":"144aff60-b302-4731-9031-da37c7aebea0","Type":"ContainerDied","Data":"445b2a58d2a5908721647d1d44f1268cebdc691456bb043d4a72a436185d8c58"}
Nov 02 18:20:20.072514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.072497    4476 scope.go:122] "RemoveContainer" containerID="445b2a58d2a5908721647d1d44f1268cebdc691456bb043d4a72a436185d8c58"
Nov 02 18:20:20.074374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.074349    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:20:20.074442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.074419    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:20:20.074478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.074448    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:20:20.127727 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:20.127699    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:21.083829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:21.083794    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" event={"ID":"144aff60-b302-4731-9031-da37c7aebea0","Type":"ContainerStarted","Data":"40699ed76f6ffba389edcb510cf8f9a35f33b417019ed0f7c2df5cffc21bce4b"}
Nov 02 18:20:21.664017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:21.663967    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:20:21.664406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:21.664314    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:20:21.664571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:21.664546    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 02 18:20:22.662987 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:22.662950    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:20:22.663603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:22.663568    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 02 18:20:23.669056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:23.669015    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:20:23.669517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:23.669079    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 02 18:20:24.662121 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:24.662085    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:20:24.662406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:24.662378    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 02 18:20:25.205387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:25.205351    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:25.664378 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:25.664341    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 02 18:20:25.664583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:25.664420    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 02 18:20:26.667140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:26.667109    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:20:26.667586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:26.667454    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:20:26.668255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:26.668234    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:20:29.060012 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:20:29.059969    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:20:30.016807 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:30.016768    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:34.783025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:34.782929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:39.491992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:39.491958    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:44.330337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:44.330295    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:49.016410 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:49.016373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:52.310833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:52.310785    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:20:52.311281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:52.310868    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:20:53.349219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.349185    4476 generic.go:356] "Generic (PLEG): container finished" podID="263b056f-8821-4407-b9cc-b4201d0a9952" containerID="2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6" exitCode=0
Nov 02 18:20:53.349804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.349259    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" event={"ID":"263b056f-8821-4407-b9cc-b4201d0a9952","Type":"ContainerDied","Data":"2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"}
Nov 02 18:20:53.349804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.349672    4476 scope.go:122] "RemoveContainer" containerID="2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"
Nov 02 18:20:53.351361 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.351336    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:20:53.351476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.351401    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:20:53.351476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.351437    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:20:53.803695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:53.803538    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:54.307049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:54.307011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:54.369467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:54.369427    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" event={"ID":"263b056f-8821-4407-b9cc-b4201d0a9952","Type":"ContainerStarted","Data":"edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087"}
Nov 02 18:20:54.882705 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:54.882667    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:20:54.882892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:54.882872    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:20:54.882937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:54.882917    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 18:20:55.815836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:55.815801    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:55.882774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:55.882743    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:20:55.882954 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:55.882798    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 18:20:56.474699 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:56.472757    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:56.687562 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:56.687528    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:56.882796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:56.882748    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:20:56.883223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:56.882813    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 18:20:57.883657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:57.883004    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:20:57.883657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:57.883067    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 18:20:58.883401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:58.883368    4476 patch_prober.go:29] interesting pod/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 02 18:20:58.883596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:58.883431    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 02 18:20:58.948831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:58.948727    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:59.096432 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:20:59.096391    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:20:59.463700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:59.463666    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:59.660788 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:59.660749    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:59.887225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:59.887192    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:20:59.887764 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:59.887741    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:20:59.888460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:20:59.888440    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:21:00.663346 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:00.663305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:00.907415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:00.907387    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:01.110824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.110776    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:01.433438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.433357    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"]
Nov 02 18:21:01.436540 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.436522    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.492169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.492130    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:21:01.492169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.492159    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:21:01.547398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.547345    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cxm\" (UniqueName: \"kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.547611 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.547458    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"]
Nov 02 18:21:01.547611 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.547569    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.547789 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.547621    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.648763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.648721    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.649120 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.649099    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.649317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.649291    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cxm\" (UniqueName: \"kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.652148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.652125    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.655088 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.655067    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.712449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.712364    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cxm\" (UniqueName: \"kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm\") pod \"maas-api-revoke-keys-e2e-autores-eb709f-bxj7d\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.753242 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.752507    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:01.779662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:01.776779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:02.160050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:02.160011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:02.178928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:02.178896    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"]
Nov 02 18:21:02.179086 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:21:02.178988    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61467de5_b20c_45b2_86c3_db6565d60b63.slice/crio-753410e73a0c9683446f6b3f26d9174d41fa0ce9f13dc312f5e46617215b2bba WatchSource:0}: Error finding container 753410e73a0c9683446f6b3f26d9174d41fa0ce9f13dc312f5e46617215b2bba: Status 404 returned error can't find the container with id 753410e73a0c9683446f6b3f26d9174d41fa0ce9f13dc312f5e46617215b2bba
Nov 02 18:21:02.579380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:02.579336    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d" event={"ID":"61467de5-b20c-45b2-86c3-db6565d60b63","Type":"ContainerStarted","Data":"ce33e512c8a8ef4bb2f33e8876f68e0b37be3e4407c5d4915587eebca7a801ea"}
Nov 02 18:21:02.579581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:02.579413    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d" event={"ID":"61467de5-b20c-45b2-86c3-db6565d60b63","Type":"ContainerStarted","Data":"753410e73a0c9683446f6b3f26d9174d41fa0ce9f13dc312f5e46617215b2bba"}
Nov 02 18:21:02.698424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:02.698365    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d" podStartSLOduration=1.6983473359999999 podStartE2EDuration="1.698347336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:02.696463366 +0000 UTC m=+3493.537069300" watchObservedRunningTime="2026-11-02 18:21:02.698347336 +0000 UTC m=+3493.538953270"
Nov 02 18:21:03.587902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:03.587869    4476 generic.go:356] "Generic (PLEG): container finished" podID="61467de5-b20c-45b2-86c3-db6565d60b63" containerID="ce33e512c8a8ef4bb2f33e8876f68e0b37be3e4407c5d4915587eebca7a801ea" exitCode=0
Nov 02 18:21:03.588362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:03.587969    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d" event={"ID":"61467de5-b20c-45b2-86c3-db6565d60b63","Type":"ContainerDied","Data":"ce33e512c8a8ef4bb2f33e8876f68e0b37be3e4407c5d4915587eebca7a801ea"}
Nov 02 18:21:04.152800 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.152760    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:04.382079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.382044    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:04.675473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.675445    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"]
Nov 02 18:21:04.685704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.685666    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.711722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.711692    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"]
Nov 02 18:21:04.739666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.739612    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-508b8f\""
Nov 02 18:21:04.755241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.755215    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-508b8f-dockercfg-xwwhm\""
Nov 02 18:21:04.786774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.786745    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:04.786905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.786867    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxqx\" (UniqueName: \"kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx\") pod \"payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.787003 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.786988    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.887518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.887485    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm\" (UniqueName: \"kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm\") pod \"61467de5-b20c-45b2-86c3-db6565d60b63\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") "
Nov 02 18:21:04.887728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.887537    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca\") pod \"61467de5-b20c-45b2-86c3-db6565d60b63\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") "
Nov 02 18:21:04.887789 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.887748    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca\") pod \"61467de5-b20c-45b2-86c3-db6565d60b63\" (UID: \"61467de5-b20c-45b2-86c3-db6565d60b63\") "
Nov 02 18:21:04.888079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.888053    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca" pod "61467de5-b20c-45b2-86c3-db6565d60b63" (UID: "61467de5-b20c-45b2-86c3-db6565d60b63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:04.888302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.888071    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxqx\" (UniqueName: \"kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx\") pod \"payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.888302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.888190    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.888489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.888450    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:21:04.889107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.889086    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.889951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.889928    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm" pod "61467de5-b20c-45b2-86c3-db6565d60b63" (UID: "61467de5-b20c-45b2-86c3-db6565d60b63"). InnerVolumeSpecName "kube-api-access-g2cxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:04.889951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.889939    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca" pod "61467de5-b20c-45b2-86c3-db6565d60b63" (UID: "61467de5-b20c-45b2-86c3-db6565d60b63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:04.945872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.945846    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxqx\" (UniqueName: \"kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx\") pod \"payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:04.990480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.990447    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2cxm\" (UniqueName: \"kubernetes.io/projected/61467de5-b20c-45b2-86c3-db6565d60b63-kube-api-access-g2cxm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:21:04.990480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:04.990478    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61467de5-b20c-45b2-86c3-db6565d60b63-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:21:05.004359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.004328    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:05.386210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.386185    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"]
Nov 02 18:21:05.387410 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:21:05.387389    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669e3f80_3ea6_4e85_af9b_55ae8f2f7ec5.slice/crio-13ac97a791b208fcd2ff4d865c3afe2e33dbe25eb010b9dd129f5bb10abe4475 WatchSource:0}: Error finding container 13ac97a791b208fcd2ff4d865c3afe2e33dbe25eb010b9dd129f5bb10abe4475: Status 404 returned error can't find the container with id 13ac97a791b208fcd2ff4d865c3afe2e33dbe25eb010b9dd129f5bb10abe4475
Nov 02 18:21:05.607433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.607404    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"
Nov 02 18:21:05.607654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.607440    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d" event={"ID":"61467de5-b20c-45b2-86c3-db6565d60b63","Type":"ContainerDied","Data":"753410e73a0c9683446f6b3f26d9174d41fa0ce9f13dc312f5e46617215b2bba"}
Nov 02 18:21:05.607654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.607491    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="753410e73a0c9683446f6b3f26d9174d41fa0ce9f13dc312f5e46617215b2bba"
Nov 02 18:21:05.609466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.609439    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" event={"ID":"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5","Type":"ContainerStarted","Data":"c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576"}
Nov 02 18:21:05.609613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.609488    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" event={"ID":"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5","Type":"ContainerStarted","Data":"13ac97a791b208fcd2ff4d865c3afe2e33dbe25eb010b9dd129f5bb10abe4475"}
Nov 02 18:21:05.609719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.609651    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:05.650096 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.649983    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:05.729950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.729902    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" podStartSLOduration=1.729887181 podStartE2EDuration="1.729887181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:05.727018293 +0000 UTC m=+3496.567624230" watchObservedRunningTime="2026-11-02 18:21:05.729887181 +0000 UTC m=+3496.570493114"
Nov 02 18:21:05.915623 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:05.915510    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:06.123153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:06.123102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:09.661190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:09.661131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:11.468255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.468220    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:11.661603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.661568    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:11.842309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.842218    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"]
Nov 02 18:21:11.842791 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.842775    4476 cpu_manager.go:414] "containerMap: removing container" podUID="61467de5-b20c-45b2-86c3-db6565d60b63" containerName="revoke-keys"
Nov 02 18:21:11.842862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.842792    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61467de5-b20c-45b2-86c3-db6565d60b63" containerName="revoke-keys"
Nov 02 18:21:11.842941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.842930    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="61467de5-b20c-45b2-86c3-db6565d60b63" containerName="revoke-keys"
Nov 02 18:21:11.845397 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.845375    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:11.904464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.904429    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"]
Nov 02 18:21:11.952210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.948373    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e4b90c-dockercfg-p7sfg\""
Nov 02 18:21:11.952210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.948476    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e4b90c\""
Nov 02 18:21:11.973402 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.973333    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:11.973977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:11.973876    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxn7m\" (UniqueName: \"kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m\") pod \"payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.075848 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.075810    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.076032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.075906    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxn7m\" (UniqueName: \"kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m\") pod \"payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.076513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.076494    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.134513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.134477    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxn7m\" (UniqueName: \"kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m\") pod \"payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.159484 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.159453    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.544879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.544090    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"]
Nov 02 18:21:12.544879 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:21:12.544175    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8329047e_c2ca_4b56_bad9_a850231f5ae1.slice/crio-a682a709b402ced4bc41dcba0f6e1357b4ec5a04fa93a40313b107742690ecab WatchSource:0}: Error finding container a682a709b402ced4bc41dcba0f6e1357b4ec5a04fa93a40313b107742690ecab: Status 404 returned error can't find the container with id a682a709b402ced4bc41dcba0f6e1357b4ec5a04fa93a40313b107742690ecab
Nov 02 18:21:12.666185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.666133    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c" event={"ID":"8329047e-c2ca-4b56-bad9-a850231f5ae1","Type":"ContainerStarted","Data":"69b4242159e7bd334e47c5e2fc534e5eaa89632c88a535483241360ff6de3f9e"}
Nov 02 18:21:12.666375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.666241    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:12.666375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.666266    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c" event={"ID":"8329047e-c2ca-4b56-bad9-a850231f5ae1","Type":"ContainerStarted","Data":"a682a709b402ced4bc41dcba0f6e1357b4ec5a04fa93a40313b107742690ecab"}
Nov 02 18:21:12.789849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:12.789797    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c" podStartSLOduration=1.789783226 podStartE2EDuration="1.789783226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:12.785763992 +0000 UTC m=+3503.626369926" watchObservedRunningTime="2026-11-02 18:21:12.789783226 +0000 UTC m=+3503.630389220"
Nov 02 18:21:14.509240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:14.509207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:16.619312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:16.619285    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:21:19.321415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:19.321382    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:20.050312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:20.050266    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"]
Nov 02 18:21:20.098959 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:20.098928    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d7e2-t88ss"]
Nov 02 18:21:21.383130 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:21.383098    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2e934a-ec45-436c-aa46-1fde5b13ac2d" path="/var/lib/kubelet/pods/5a2e934a-ec45-436c-aa46-1fde5b13ac2d/volumes"
Nov 02 18:21:23.677355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:23.677316    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:21:24.144533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:24.144485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:27.769143 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:27.769108    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:27.977320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:27.977274    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:29.060704 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:29.060665    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:21:29.082057 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:29.082027    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:29.278282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:29.278249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:29.779124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:29.779085    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:30.005086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:30.005051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:30.053494 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:30.053412    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"]
Nov 02 18:21:30.102512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:30.102481    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268c9a78-d2b55"]
Nov 02 18:21:31.384992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:31.384957    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d70b6c5-5e72-4498-a33a-08b6a7a925b8" path="/var/lib/kubelet/pods/4d70b6c5-5e72-4498-a33a-08b6a7a925b8/volumes"
Nov 02 18:21:33.496935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:33.496898    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:34.572559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:34.572518    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:34.780881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:34.780847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:35.535008 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:35.534976    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:35.740889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:35.740860    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"]
Nov 02 18:21:35.745931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:35.745906    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eb709f-bxj7d"]
Nov 02 18:21:35.817583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:35.817514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:36.092818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.092699    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:36.526169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.526126    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7523-cb4035"]
Nov 02 18:21:36.529750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.529724    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:36.583430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.583394    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7523-cb4035"]
Nov 02 18:21:36.628116 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.628090    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4gwc\""
Nov 02 18:21:36.664910 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.664873    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6cn\" (UniqueName: \"kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn\") pod \"test-curl-7523-cb4035\" (UID: \"43a9be39-c593-4a53-9618-0b9497c72b96\") " pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:36.765787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.765753    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6cn\" (UniqueName: \"kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn\") pod \"test-curl-7523-cb4035\" (UID: \"43a9be39-c593-4a53-9618-0b9497c72b96\") " pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:36.823169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.823097    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6cn\" (UniqueName: \"kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn\") pod \"test-curl-7523-cb4035\" (UID: \"43a9be39-c593-4a53-9618-0b9497c72b96\") " pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:36.841852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.841821    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:36.876997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:36.876966    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:37.231947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.231912    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7523-cb4035"]
Nov 02 18:21:37.234184 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:21:37.234156    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43a9be39_c593_4a53_9618_0b9497c72b96.slice/crio-102f1ee9f9c65d3221d664d05a036d3659d20dc48abaa7e335048839ae27069e WatchSource:0}: Error finding container 102f1ee9f9c65d3221d664d05a036d3659d20dc48abaa7e335048839ae27069e: Status 404 returned error can't find the container with id 102f1ee9f9c65d3221d664d05a036d3659d20dc48abaa7e335048839ae27069e
Nov 02 18:21:37.384179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.384147    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61467de5-b20c-45b2-86c3-db6565d60b63" path="/var/lib/kubelet/pods/61467de5-b20c-45b2-86c3-db6565d60b63/volumes"
Nov 02 18:21:37.683324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.683286    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:37.865295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.865251    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7523-cb4035" event={"ID":"43a9be39-c593-4a53-9618-0b9497c72b96","Type":"ContainerStarted","Data":"6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368"}
Nov 02 18:21:37.865830 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.865306    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7523-cb4035" event={"ID":"43a9be39-c593-4a53-9618-0b9497c72b96","Type":"ContainerStarted","Data":"102f1ee9f9c65d3221d664d05a036d3659d20dc48abaa7e335048839ae27069e"}
Nov 02 18:21:37.902664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.902614    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:37.981054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:37.980931    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7523-cb4035" podStartSLOduration=1.980914226 podStartE2EDuration="1.980914226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:37.979658827 +0000 UTC m=+3528.820264758" watchObservedRunningTime="2026-11-02 18:21:37.980914226 +0000 UTC m=+3528.821520163"
Nov 02 18:21:38.105500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:38.105460    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:39.233245 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.233205    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:39.375654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.375116    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7523-cb4035"]
Nov 02 18:21:39.393932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.393897    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7523-cb4035"]
Nov 02 18:21:39.435572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.435538    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:39.652456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.652414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:39.846360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.846314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:39.882321 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:39.882274    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7523-cb4035" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" containerName="test-curl-7523-cb4035" containerID="cri-o://6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368" gracePeriod=2
Nov 02 18:21:39.934134 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:39.934051    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7523-cb4035\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:40.001904 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.001867    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"]
Nov 02 18:21:40.002570 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.002544    4476 cpu_manager.go:414] "containerMap: removing container" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" containerName="test-curl-7523-cb4035"
Nov 02 18:21:40.002785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.002758    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" containerName="test-curl-7523-cb4035"
Nov 02 18:21:40.002933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.002918    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" containerName="test-curl-7523-cb4035"
Nov 02 18:21:40.006141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.006125    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.057380 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:40.057338    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7523-cb4035\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:40.075201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.074920    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a72f55\""
Nov 02 18:21:40.075201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.075106    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a72f55-dockercfg-p2cz8\""
Nov 02 18:21:40.077271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.077243    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"]
Nov 02 18:21:40.106262 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.106231    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.106464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.106288    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjzl\" (UniqueName: \"kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl\") pod \"payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.207505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.207410    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.207505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.207471    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjzl\" (UniqueName: \"kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl\") pod \"payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.208464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.208434    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.266596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.266565    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjzl\" (UniqueName: \"kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl\") pod \"payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.321301 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.321274    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.511396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.510314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:40.727670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.717808    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"]
Nov 02 18:21:40.727670 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:21:40.719114    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb24fbe_d20c_4690_ae54_b564288a80b7.slice/crio-1323cf89b421258cb1cfdb770e6316bc9d5aa87d8355c497d3c03704c7ea7f62 WatchSource:0}: Error finding container 1323cf89b421258cb1cfdb770e6316bc9d5aa87d8355c497d3c03704c7ea7f62: Status 404 returned error can't find the container with id 1323cf89b421258cb1cfdb770e6316bc9d5aa87d8355c497d3c03704c7ea7f62
Nov 02 18:21:40.891579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.891546    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7" event={"ID":"5bb24fbe-d20c-4690-ae54-b564288a80b7","Type":"ContainerStarted","Data":"7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f"}
Nov 02 18:21:40.891816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.891601    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7" event={"ID":"5bb24fbe-d20c-4690-ae54-b564288a80b7","Type":"ContainerStarted","Data":"1323cf89b421258cb1cfdb770e6316bc9d5aa87d8355c497d3c03704c7ea7f62"}
Nov 02 18:21:40.891816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.891688    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:40.966048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:40.966011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:41.018141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:41.018037    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7" podStartSLOduration=2.018022938 podStartE2EDuration="2.018022938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:41.014417667 +0000 UTC m=+3531.855023593" watchObservedRunningTime="2026-11-02 18:21:41.018022938 +0000 UTC m=+3531.858628872"
Nov 02 18:21:41.065368 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:41.065324    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7523-cb4035\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:41.733765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:41.733732    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:42.051124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.051013    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:42.151796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.151769    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:42.333472 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.333368    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn\" (UniqueName: \"kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn\") pod \"43a9be39-c593-4a53-9618-0b9497c72b96\" (UID: \"43a9be39-c593-4a53-9618-0b9497c72b96\") "
Nov 02 18:21:42.335728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.335696    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn" pod "43a9be39-c593-4a53-9618-0b9497c72b96" (UID: "43a9be39-c593-4a53-9618-0b9497c72b96"). InnerVolumeSpecName "kube-api-access-cj6cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:42.434953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.434909    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6cn\" (UniqueName: \"kubernetes.io/projected/43a9be39-c593-4a53-9618-0b9497c72b96-kube-api-access-cj6cn\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:21:42.909722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.909680    4476 generic.go:356] "Generic (PLEG): container finished" podID="43a9be39-c593-4a53-9618-0b9497c72b96" containerID="6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368" exitCode=137
Nov 02 18:21:42.910236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.909742    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:42.910236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.909779    4476 scope.go:122] "RemoveContainer" containerID="6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368"
Nov 02 18:21:42.920112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.920088    4476 scope.go:122] "RemoveContainer" containerID="6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368"
Nov 02 18:21:42.920396 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:42.920369    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368\": container with ID starting with 6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368 not found: ID does not exist" containerID="6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368"
Nov 02 18:21:42.920455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:42.920407    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368"} err="failed to get container status \"6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368\": rpc error: code = NotFound desc = could not find container \"6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368\": container with ID starting with 6fb172c29b2a4f150b7d11c928fcf741b80174314bae55993d2c1ab2e33a6368 not found: ID does not exist"
Nov 02 18:21:42.969396 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:42.969352    4476 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7523-cb4035\" is forbidden: User \"system:node:ip-10-0-136-192.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-192.us-west-1.compute.internal' and this object" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" pod="opendatahub/test-curl-7523-cb4035"
Nov 02 18:21:43.385517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:43.385469    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a9be39-c593-4a53-9618-0b9497c72b96" path="/var/lib/kubelet/pods/43a9be39-c593-4a53-9618-0b9497c72b96/volumes"
Nov 02 18:21:43.830806 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:43.830729    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:44.047099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:44.047070    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:44.249427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:44.249392    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:45.624563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:45.624527    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:47.120218 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:47.120182    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:47.333112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:47.327454    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:47.672961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:47.672921    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:50.782591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:50.782536    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:51.905179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:51.905152    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:21:52.136382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:52.136348    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:52.336345 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:52.336259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:55.541474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:55.541402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:59.061174 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:21:59.061132    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:21:59.933642 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:21:59.933585    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:00.268845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:00.268770    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:05.107913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:05.107877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:09.858174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:09.858135    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:14.585736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:14.585702    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:19.308967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:19.308935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:23.664738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.664707    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"]
Nov 02 18:22:23.668687 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.668668    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.707619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.707590    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.707811 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.707790    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.707870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.707843    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4tv\" (UniqueName: \"kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.707953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.707931    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.721146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.721112    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-508b8f-serving-cert\""
Nov 02 18:22:23.774505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.774474    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"]
Nov 02 18:22:23.809138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.809106    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.809320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.809175    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4tv\" (UniqueName: \"kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.809320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.809225    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.809320 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:22:23.809281    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-serving-cert: secret "maas-api-e2e-extmod-a-508b8f-serving-cert" not found
Nov 02 18:22:23.809320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.809299    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.809555 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:22:23.809365    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls podName:f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9 nodeName:}" failed. No retries permitted until 2026-11-02 18:22:24.309343314 +0000 UTC m=+3575.149949244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls") pod "maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" (UID: "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9") : secret "maas-api-e2e-extmod-a-508b8f-serving-cert" not found
Nov 02 18:22:23.810429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.810398    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.812245 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.812221    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.868539 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.868513    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4tv\" (UniqueName: \"kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:23.913008 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:23.912968    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:24.104391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:24.104306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:24.317579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:24.317541    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:24.320142 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:24.320113    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:24.583117 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:24.583087    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:24.966435 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:24.966405    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"]
Nov 02 18:22:24.967103 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:22:24.967081    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32bc4e6_d62f_4524_aaf0_0e3fda3b11c9.slice/crio-1d783d29cbc0f270c0991fa0c961aac6da4774bbd49346ec28107927dee4787f WatchSource:0}: Error finding container 1d783d29cbc0f270c0991fa0c961aac6da4774bbd49346ec28107927dee4787f: Status 404 returned error can't find the container with id 1d783d29cbc0f270c0991fa0c961aac6da4774bbd49346ec28107927dee4787f
Nov 02 18:22:25.218271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:25.218192    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" event={"ID":"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9","Type":"ContainerStarted","Data":"1d783d29cbc0f270c0991fa0c961aac6da4774bbd49346ec28107927dee4787f"}
Nov 02 18:22:26.227150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:26.227054    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" event={"ID":"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9","Type":"ContainerStarted","Data":"e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59"}
Nov 02 18:22:26.227150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:26.227129    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:26.347078 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:26.347027    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" podStartSLOduration=2.437891139 podStartE2EDuration="3.347013505s" totalImagesPullingTime="909.122366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:26.343876268 +0000 UTC m=+3577.184482202" watchObservedRunningTime="2026-11-02 18:22:26.347013505 +0000 UTC m=+3577.187619457"
Nov 02 18:22:28.888183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:28.888146    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:29.059081 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:22:29.059043    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:22:32.242149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:32.242118    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:22:33.383854 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.383821    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"]
Nov 02 18:22:33.387978 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.387956    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.439970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.439932    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e4b90c-serving-cert\""
Nov 02 18:22:33.493446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.493409    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"]
Nov 02 18:22:33.529722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.529688    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.529916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.529807    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlw5s\" (UniqueName: \"kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.529993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.529942    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.530053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.530029    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.631774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.631736    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.632217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.632192    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.632651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.632605    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.632901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.632864    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlw5s\" (UniqueName: \"kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.633272 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:22:33.632087    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-serving-cert: secret "maas-api-e2e-extmod-b-e4b90c-serving-cert" not found
Nov 02 18:22:33.633371 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:22:33.633323    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls podName:9b55cd26-1b6d-42ba-925d-84f49ab4304e nodeName:}" failed. No retries permitted until 2026-11-02 18:22:34.133301548 +0000 UTC m=+3584.973907474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls") pod "maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" (UID: "9b55cd26-1b6d-42ba-925d-84f49ab4304e") : secret "maas-api-e2e-extmod-b-e4b90c-serving-cert" not found
Nov 02 18:22:33.633514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.633491    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.638395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.638329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:33.643603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.643533    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.708511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.708477    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlw5s\" (UniqueName: \"kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:33.832511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:33.832477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:34.020530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:34.020436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:34.140110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:34.140071    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:34.142906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:34.142881    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:34.301873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:34.301797    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:34.692100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:34.692074    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"]
Nov 02 18:22:34.693250 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:22:34.693225    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b55cd26_1b6d_42ba_925d_84f49ab4304e.slice/crio-a606242e7fd3543c25bafc2948c1e67e45d3bc7ec40652fab659a9248ef8ea4c WatchSource:0}: Error finding container a606242e7fd3543c25bafc2948c1e67e45d3bc7ec40652fab659a9248ef8ea4c: Status 404 returned error can't find the container with id a606242e7fd3543c25bafc2948c1e67e45d3bc7ec40652fab659a9248ef8ea4c
Nov 02 18:22:35.296943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:35.296904    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" event={"ID":"9b55cd26-1b6d-42ba-925d-84f49ab4304e","Type":"ContainerStarted","Data":"a606242e7fd3543c25bafc2948c1e67e45d3bc7ec40652fab659a9248ef8ea4c"}
Nov 02 18:22:36.306748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:36.306699    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" event={"ID":"9b55cd26-1b6d-42ba-925d-84f49ab4304e","Type":"ContainerStarted","Data":"1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69"}
Nov 02 18:22:36.307149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:36.306815    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:36.425168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:36.423923    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" podStartSLOduration=2.563119127 podStartE2EDuration="3.423903718s" totalImagesPullingTime="860.784591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:36.42268215 +0000 UTC m=+3587.263288085" watchObservedRunningTime="2026-11-02 18:22:36.423903718 +0000 UTC m=+3587.264509655"
Nov 02 18:22:38.571103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:38.571057    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:40.050506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:40.050470    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"]
Nov 02 18:22:40.100601 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:40.100567    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17af1101-p52wl"]
Nov 02 18:22:41.385569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:41.385535    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872dfecd-04fe-4472-a211-7b50b921dcb1" path="/var/lib/kubelet/pods/872dfecd-04fe-4472-a211-7b50b921dcb1/volumes"
Nov 02 18:22:42.318949 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:42.318920    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:22:43.362514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:43.362479    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:48.048141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:48.048084    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"]
Nov 02 18:22:48.098440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:48.098405    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80f2595e-jz6vd"]
Nov 02 18:22:48.107121 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:48.107091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:49.387031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:49.386993    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6378f1c0-7559-4bb1-85aa-262a96899475" path="/var/lib/kubelet/pods/6378f1c0-7559-4bb1-85aa-262a96899475/volumes"
Nov 02 18:22:50.006150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:50.006115    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:22:50.013028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:50.013004    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:22:50.885519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:50.885488    4476 scope.go:122] "RemoveContainer" containerID="f760925d7bac74d74b67d8038a555a9d1cacaf2c84ac20cfa4a78046a18cefb2"
Nov 02 18:22:50.895751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:50.895728    4476 scope.go:122] "RemoveContainer" containerID="09cd49c6934cc37562128711635202df259225d43764ecb90b332f8589af0309"
Nov 02 18:22:51.047815 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:51.047757    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"]
Nov 02 18:22:51.096984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:51.096956    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec263b49-6ljk7"]
Nov 02 18:22:51.385559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:51.385518    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4385c942-66a9-4242-be1b-04623d65699d" path="/var/lib/kubelet/pods/4385c942-66a9-4242-be1b-04623d65699d/volumes"
Nov 02 18:22:52.842490 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:52.842448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:57.519581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:57.519546    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:59.059062 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:22:59.059020    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:22:59.063610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:59.063584    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"]
Nov 02 18:22:59.113314 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:59.113277    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495c8ff8-p6bhl"]
Nov 02 18:22:59.384148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:22:59.384106    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3400ce-7a44-4e1e-84c1-7f0f732dc273" path="/var/lib/kubelet/pods/bc3400ce-7a44-4e1e-84c1-7f0f732dc273/volumes"
Nov 02 18:23:00.300648 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:00.300584    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:02.331979 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:02.331935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:03.447165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:03.447128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:03.643405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:03.643367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:06.927604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:06.927559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:07.125034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:07.124993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:11.902434 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:11.902393    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:16.712502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:16.712464    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:21.413735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:21.413702    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:21.596653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:21.596590    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:21.779228 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:21.779144    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:24.929393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:24.929358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:25.144693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:25.144661    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:26.317871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:26.317839    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:26.534844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:26.534810    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:29.060386 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:23:29.060345    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:23:31.178934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:31.178822    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:35.911514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:35.911469    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:40.710682 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:40.710643    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:45.587665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:45.587594    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:50.512776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:50.512739    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:50.926146 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:50.926119    4476 scope.go:122] "RemoveContainer" containerID="2da6f2fd091ab75c358f89260e7069ef12f5c72535203063d4000756d196bccc"
Nov 02 18:23:50.936483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:50.936464    4476 scope.go:122] "RemoveContainer" containerID="1a3c175064c19b17ef31b534ba6455b2e51c6ecd09b1b63de6bea3bd3559551c"
Nov 02 18:23:50.949491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:50.949470    4476 scope.go:122] "RemoveContainer" containerID="b7bbf90ad2a3f2483091099770066fa45a3c5417a6522724481e99a9cc06d2bd"
Nov 02 18:23:55.599931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:23:55.599886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:59.059204 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:23:59.059167    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:24:00.446066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:00.446011    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:00.650816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:00.650773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:04.857264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:04.857228    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:06.212252 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:06.212220    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:09.460206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:09.460174    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:09.870493 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:09.870460    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:11.323231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:11.323197    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:11.392602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:11.392572    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"]
Nov 02 18:24:11.835666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:11.835597    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:12.024815 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:12.024784    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:14.588834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:14.588800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:15.828179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:15.828146    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:16.392898 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.392861    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" podUID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" containerName="payload-processing" containerID="cri-o://c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576" gracePeriod=25
Nov 02 18:24:16.674469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.674430    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:16.704049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.704018    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:24:16.752362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.752336    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume\") pod \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") "
Nov 02 18:24:16.752536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.752417    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx\" (UniqueName: \"kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx\") pod \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\" (UID: \"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5\") "
Nov 02 18:24:16.752816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.752791    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume" pod "669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" (UID: "669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:16.754601 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.754585    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx" pod "669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" (UID: "669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5"). InnerVolumeSpecName "kube-api-access-ctxqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:16.854312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.854272    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:16.854312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.854305    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxqx\" (UniqueName: \"kubernetes.io/projected/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5-kube-api-access-ctxqx\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:16.869048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:16.869023    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:17.100958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.100872    4476 generic.go:356] "Generic (PLEG): container finished" podID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" containerID="c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576" exitCode=0
Nov 02 18:24:17.100958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.100923    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"
Nov 02 18:24:17.101169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.100958    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" event={"ID":"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5","Type":"ContainerDied","Data":"c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576"}
Nov 02 18:24:17.101169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.101009    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" event={"ID":"669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5","Type":"ContainerDied","Data":"13ac97a791b208fcd2ff4d865c3afe2e33dbe25eb010b9dd129f5bb10abe4475"}
Nov 02 18:24:17.101169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.101032    4476 scope.go:122] "RemoveContainer" containerID="c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576"
Nov 02 18:24:17.115170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.115156    4476 scope.go:122] "RemoveContainer" containerID="c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576"
Nov 02 18:24:17.115439 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:24:17.115416    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576\": container with ID starting with c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576 not found: ID does not exist" containerID="c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576"
Nov 02 18:24:17.115513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.115450    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576"} err="failed to get container status \"c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576\": rpc error: code = NotFound desc = could not find container \"c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576\": container with ID starting with c6da4f0aaf7b6dd13dc29b1b1220be7e98a791295dbe6c12f4dac7bc28059576 not found: ID does not exist"
Nov 02 18:24:17.203108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.203068    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"]
Nov 02 18:24:17.203317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.203296    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" containerID="cri-o://0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f" gracePeriod=30
Nov 02 18:24:17.280206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.280176    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"]
Nov 02 18:24:17.302815 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.302789    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl"]
Nov 02 18:24:17.382689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.382656    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" path="/var/lib/kubelet/pods/669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5/volumes"
Nov 02 18:24:17.617837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.617800    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-508b8f-78dd585d69-5b2gl" podUID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.194:50052\" within 1s: context deadline exceeded"
Nov 02 18:24:17.618026 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:24:17.617891    4476 logging.go:55] [core] [Channel #6571 SubChannel #6572]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.194:50052", ServerName: "10.133.0.194:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.194:50052: operation was canceled"
Nov 02 18:24:17.880250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:17.880215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:19.843570 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:19.843532    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:21.823705 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:21.823668    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:22.082408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.082317    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:22.126297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.126261    4476 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:24:22.126468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.126314    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:24:22.491791 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.491764    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:24:22.624966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.624929    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.624979    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625017    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625043    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625081    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625107    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625185    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625218    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625288    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data\") pod \"e4d48148-8479-4881-b482-092c1a7920ca\" (UID: \"e4d48148-8479-4881-b482-092c1a7920ca\") "
Nov 02 18:24:22.625487 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625288    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:22.625754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625547    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:22.626014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.625976    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.626014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.626001    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d48148-8479-4881-b482-092c1a7920ca-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.626170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.626115    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:22.626396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.626222    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:22.626542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.626510    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:22.627714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.627693    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "kube-api-access-j7kzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:22.627864 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.627835    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:24:22.627997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.627979    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:24:22.628219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.628199    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token" pod "e4d48148-8479-4881-b482-092c1a7920ca" (UID: "e4d48148-8479-4881-b482-092c1a7920ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:22.726916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726884    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.726916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726911    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7kzm\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-kube-api-access-j7kzm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.726916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726920    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.727148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726929    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d48148-8479-4881-b482-092c1a7920ca-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.727148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726939    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d48148-8479-4881-b482-092c1a7920ca-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.727148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726948    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:22.727148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:22.726956    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d48148-8479-4881-b482-092c1a7920ca-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:23.146840 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.146798    4476 generic.go:356] "Generic (PLEG): container finished" podID="e4d48148-8479-4881-b482-092c1a7920ca" containerID="0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f" exitCode=0
Nov 02 18:24:23.146840 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.146841    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"
Nov 02 18:24:23.147338 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.146884    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" event={"ID":"e4d48148-8479-4881-b482-092c1a7920ca","Type":"ContainerDied","Data":"0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f"}
Nov 02 18:24:23.147338 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.146941    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5" event={"ID":"e4d48148-8479-4881-b482-092c1a7920ca","Type":"ContainerDied","Data":"390d94f2690a97be4be82178569695a1a3d414dcec73f115f72ec7bfe04c7a9d"}
Nov 02 18:24:23.147338 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.146968    4476 scope.go:122] "RemoveContainer" containerID="0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f"
Nov 02 18:24:23.158580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.158561    4476 scope.go:122] "RemoveContainer" containerID="0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f"
Nov 02 18:24:23.158905 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:24:23.158883    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f\": container with ID starting with 0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f not found: ID does not exist" containerID="0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f"
Nov 02 18:24:23.159016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.158910    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f"} err="failed to get container status \"0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f\": rpc error: code = NotFound desc = could not find container \"0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f\": container with ID starting with 0fd1b8b6fb93b321b4eb86201bfa733b59ab659a699ef6f64fdc8011027f683f not found: ID does not exist"
Nov 02 18:24:23.314465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.314423    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"]
Nov 02 18:24:23.363873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.363838    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-j6zw5"]
Nov 02 18:24:23.383987 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:23.383945    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d48148-8479-4881-b482-092c1a7920ca" path="/var/lib/kubelet/pods/e4d48148-8479-4881-b482-092c1a7920ca/volumes"
Nov 02 18:24:25.094110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:25.094066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:29.060248 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:24:29.060208    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:24:30.395858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:30.395824    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:33.357561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.357525    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"]
Nov 02 18:24:33.358131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.358117    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy"
Nov 02 18:24:33.358174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.358134    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy"
Nov 02 18:24:33.358174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.358143    4476 cpu_manager.go:414] "containerMap: removing container" podUID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" containerName="payload-processing"
Nov 02 18:24:33.358174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.358148    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" containerName="payload-processing"
Nov 02 18:24:33.358270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.358242    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d48148-8479-4881-b482-092c1a7920ca" containerName="istio-proxy"
Nov 02 18:24:33.358270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.358252    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="669e3f80-3ea6-4e85-af9b-55ae8f2f7ec5" containerName="payload-processing"
Nov 02 18:24:33.361676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.361660    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.445256 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.445225    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e4b90c-openshift-default-dockercfg-6kx9n\""
Nov 02 18:24:33.464391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464343    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464389    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8t2c\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464454    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464499    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464542    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464591    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464621    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464683    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.464953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.464758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.467658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.467601    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"]
Nov 02 18:24:33.566345 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566302    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566345 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566352    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566380    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566400    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566441    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566520    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566546    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8t2c\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566618    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566670    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.566926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.566873    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.567114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.567082    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.567264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.567160    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.567455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.567429    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.567645 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.567600    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.569262 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.569241    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.569411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.569392    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.624389 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.624352    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8t2c\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.625299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.625278    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:33.677814 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:33.677786    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:34.115725 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.115699    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"]
Nov 02 18:24:34.116788 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:24:34.116758    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6e0939_d38e_477a_86a8_a6218438a142.slice/crio-0ee5e6954f14e69a09bd6ce6417de143f5bfe921a08b42e02f5fa4f6066478eb WatchSource:0}: Error finding container 0ee5e6954f14e69a09bd6ce6417de143f5bfe921a08b42e02f5fa4f6066478eb: Status 404 returned error can't find the container with id 0ee5e6954f14e69a09bd6ce6417de143f5bfe921a08b42e02f5fa4f6066478eb
Nov 02 18:24:34.119349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.119319    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:24:34.119400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.119385    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:24:34.119442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.119412    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:24:34.265653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.265580    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" event={"ID":"7c6e0939-d38e-477a-86a8-a6218438a142","Type":"ContainerStarted","Data":"80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9"}
Nov 02 18:24:34.265853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.265661    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" event={"ID":"7c6e0939-d38e-477a-86a8-a6218438a142","Type":"ContainerStarted","Data":"0ee5e6954f14e69a09bd6ce6417de143f5bfe921a08b42e02f5fa4f6066478eb"}
Nov 02 18:24:34.391724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.389806    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podStartSLOduration=1.3897895120000001 podStartE2EDuration="1.389789512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:24:34.389075583 +0000 UTC m=+3705.229681516" watchObservedRunningTime="2026-11-02 18:24:34.389789512 +0000 UTC m=+3705.230395446"
Nov 02 18:24:34.678421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:34.678336    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:35.184363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:35.184333    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:35.679981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:35.679942    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 18:24:35.680421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:35.680004    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 18:24:36.679124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:36.679087    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 18:24:36.679326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:36.679145    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 18:24:37.678152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:37.678115    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 18:24:37.678578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:37.678176    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 18:24:38.002697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:38.002211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:38.679646 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:38.679034    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 18:24:38.679646 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:38.679091    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 18:24:39.678165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:39.678131    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 18:24:39.678384 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:39.678190    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 18:24:40.024121 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:40.023982    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:40.683245 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:40.683211    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:40.683699 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:40.683675    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:40.684285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:40.684269    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:24:44.019071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:44.019030    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:44.398986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:44.398954    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"]
Nov 02 18:24:44.512822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:44.512785    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"]
Nov 02 18:24:44.836216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:44.836114    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:45.047297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:45.047242    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:47.145498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:47.145446    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:49.399859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.399803    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh" podUID="bc997153-2a84-43eb-9d52-3c2f00205680" containerName="payload-processing" containerID="cri-o://08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37" gracePeriod=25
Nov 02 18:24:49.513263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.513203    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v" podUID="e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" containerName="payload-pre-processing" containerID="cri-o://1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a" gracePeriod=25
Nov 02 18:24:49.679547 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.679513    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:24:49.681042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.681014    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:49.784526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.784496    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume\") pod \"bc997153-2a84-43eb-9d52-3c2f00205680\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") "
Nov 02 18:24:49.784746 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.784559    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb\" (UniqueName: \"kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb\") pod \"bc997153-2a84-43eb-9d52-3c2f00205680\" (UID: \"bc997153-2a84-43eb-9d52-3c2f00205680\") "
Nov 02 18:24:49.784951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.784928    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume" pod "bc997153-2a84-43eb-9d52-3c2f00205680" (UID: "bc997153-2a84-43eb-9d52-3c2f00205680"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:49.785090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.785073    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc997153-2a84-43eb-9d52-3c2f00205680-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:49.787270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.787152    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb" pod "bc997153-2a84-43eb-9d52-3c2f00205680" (UID: "bc997153-2a84-43eb-9d52-3c2f00205680"). InnerVolumeSpecName "kube-api-access-2vpzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:49.805649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.805607    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:24:49.886350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.886306    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vpzb\" (UniqueName: \"kubernetes.io/projected/bc997153-2a84-43eb-9d52-3c2f00205680-kube-api-access-2vpzb\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:49.987488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.987399    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume\") pod \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") "
Nov 02 18:24:49.987488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.987439    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd\" (UniqueName: \"kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd\") pod \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\" (UID: \"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413\") "
Nov 02 18:24:49.987917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.987893    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume" pod "e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" (UID: "e8cd8953-99d2-44d4-9a0c-e4cf14e1d413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:49.988132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.988117    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:49.989952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:49.989920    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd" pod "e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" (UID: "e8cd8953-99d2-44d4-9a0c-e4cf14e1d413"). InnerVolumeSpecName "kube-api-access-wd7zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:50.040208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.040172    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:50.089618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.089582    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7zd\" (UniqueName: \"kubernetes.io/projected/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413-kube-api-access-wd7zd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:50.233061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.233028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:50.392885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.392851    4476 generic.go:356] "Generic (PLEG): container finished" podID="e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" containerID="1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a" exitCode=0
Nov 02 18:24:50.393042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.392899    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"
Nov 02 18:24:50.393042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.392936    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v" event={"ID":"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413","Type":"ContainerDied","Data":"1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a"}
Nov 02 18:24:50.393042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.392992    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v" event={"ID":"e8cd8953-99d2-44d4-9a0c-e4cf14e1d413","Type":"ContainerDied","Data":"5b9b43c6c2c072dff3354237103e2d2c2be53e758c95d95a4c35d1c68c7634f8"}
Nov 02 18:24:50.393042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.393017    4476 scope.go:122] "RemoveContainer" containerID="1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a"
Nov 02 18:24:50.394760 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.394739    4476 generic.go:356] "Generic (PLEG): container finished" podID="bc997153-2a84-43eb-9d52-3c2f00205680" containerID="08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37" exitCode=0
Nov 02 18:24:50.394918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.394787    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh" event={"ID":"bc997153-2a84-43eb-9d52-3c2f00205680","Type":"ContainerDied","Data":"08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37"}
Nov 02 18:24:50.394918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.394792    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"
Nov 02 18:24:50.394918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.394826    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh" event={"ID":"bc997153-2a84-43eb-9d52-3c2f00205680","Type":"ContainerDied","Data":"f55ceda009de56ddcb1ac949c9d4c28baa99f751df5e6b6ff015795aa4de2796"}
Nov 02 18:24:50.405066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.404608    4476 scope.go:122] "RemoveContainer" containerID="1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a"
Nov 02 18:24:50.405066 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:24:50.404964    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a\": container with ID starting with 1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a not found: ID does not exist" containerID="1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a"
Nov 02 18:24:50.405066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.405006    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a"} err="failed to get container status \"1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a\": rpc error: code = NotFound desc = could not find container \"1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a\": container with ID starting with 1db9f7caf1d72a039a1a87548c308b0d3d90613f9ecefaaa10090a64a93dda1a not found: ID does not exist"
Nov 02 18:24:50.405066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.405028    4476 scope.go:122] "RemoveContainer" containerID="08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37"
Nov 02 18:24:50.424207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.424186    4476 scope.go:122] "RemoveContainer" containerID="08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37"
Nov 02 18:24:50.424481 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:24:50.424463    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37\": container with ID starting with 08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37 not found: ID does not exist" containerID="08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37"
Nov 02 18:24:50.424531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.424489    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37"} err="failed to get container status \"08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37\": rpc error: code = NotFound desc = could not find container \"08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37\": container with ID starting with 08cbf9caeef76368d6e6f8c29adcb6fee5851085a7ac00c8ed80d3cdb0d2da37 not found: ID does not exist"
Nov 02 18:24:50.561715 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.561679    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"]
Nov 02 18:24:50.609017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.608982    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a9496c-75558c55f-pkv5v"]
Nov 02 18:24:50.717813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.717712    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"]
Nov 02 18:24:50.722700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.722669    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a9496c-59fc6fff59-7lrwh"]
Nov 02 18:24:50.993505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:50.993414    4476 scope.go:122] "RemoveContainer" containerID="67c57a7bcf54dee122a2a829dbeb7fb0b70e89605f400ded0217c60e69ec60c4"
Nov 02 18:24:51.384738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:51.384703    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc997153-2a84-43eb-9d52-3c2f00205680" path="/var/lib/kubelet/pods/bc997153-2a84-43eb-9d52-3c2f00205680/volumes"
Nov 02 18:24:51.385138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:51.385124    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" path="/var/lib/kubelet/pods/e8cd8953-99d2-44d4-9a0c-e4cf14e1d413/volumes"
Nov 02 18:24:52.287158 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:52.287121    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"]
Nov 02 18:24:52.769549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:52.769514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:52.966810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:52.966774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:53.930174 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:53.930142    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:55.005260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:55.005230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:56.371976 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:56.371939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:57.086481 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.086444    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:57.288258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.288218    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn" podUID="0efe75f2-184b-4ff5-bbf2-7791c96d9417" containerName="payload-pre-processing" containerID="cri-o://15c3eaf70b514a497ac0f524db58ccb9930a010d1e18ac50016c9b4f8dd4f88d" gracePeriod=25
Nov 02 18:24:57.298171 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.298143    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:57.448495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.448458    4476 generic.go:356] "Generic (PLEG): container finished" podID="0efe75f2-184b-4ff5-bbf2-7791c96d9417" containerID="15c3eaf70b514a497ac0f524db58ccb9930a010d1e18ac50016c9b4f8dd4f88d" exitCode=0
Nov 02 18:24:57.448905 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.448539    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn" event={"ID":"0efe75f2-184b-4ff5-bbf2-7791c96d9417","Type":"ContainerDied","Data":"15c3eaf70b514a497ac0f524db58ccb9930a010d1e18ac50016c9b4f8dd4f88d"}
Nov 02 18:24:57.564392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.564366    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:24:57.661574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.661487    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume\") pod \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") "
Nov 02 18:24:57.661574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.661545    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm\" (UniqueName: \"kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm\") pod \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\" (UID: \"0efe75f2-184b-4ff5-bbf2-7791c96d9417\") "
Nov 02 18:24:57.661938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.661915    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume" pod "0efe75f2-184b-4ff5-bbf2-7791c96d9417" (UID: "0efe75f2-184b-4ff5-bbf2-7791c96d9417"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:57.663788 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.663769    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm" pod "0efe75f2-184b-4ff5-bbf2-7791c96d9417" (UID: "0efe75f2-184b-4ff5-bbf2-7791c96d9417"). InnerVolumeSpecName "kube-api-access-fr2xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:57.763227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.763198    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0efe75f2-184b-4ff5-bbf2-7791c96d9417-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:57.763227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:57.763225    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2xm\" (UniqueName: \"kubernetes.io/projected/0efe75f2-184b-4ff5-bbf2-7791c96d9417-kube-api-access-fr2xm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:24:58.456942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:58.456911    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"
Nov 02 18:24:58.457379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:58.456913    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn" event={"ID":"0efe75f2-184b-4ff5-bbf2-7791c96d9417","Type":"ContainerDied","Data":"dc7058b995259057a66e2931882cb343c220c4046b77992c0e1865e766c3be74"}
Nov 02 18:24:58.457379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:58.457057    4476 scope.go:122] "RemoveContainer" containerID="15c3eaf70b514a497ac0f524db58ccb9930a010d1e18ac50016c9b4f8dd4f88d"
Nov 02 18:24:58.622535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:58.622502    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"]
Nov 02 18:24:58.672037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:58.672007    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cd738b-8558c774d7-cg7nn"]
Nov 02 18:24:59.058704 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:24:59.058672    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:24:59.385464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:59.385435    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0efe75f2-184b-4ff5-bbf2-7791c96d9417" path="/var/lib/kubelet/pods/0efe75f2-184b-4ff5-bbf2-7791c96d9417/volumes"
Nov 02 18:24:59.517988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:24:59.517939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:00.254513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:00.254481    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:00.987565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:00.987524    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:01.971953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:01.971908    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:03.606253 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606218    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"]
Nov 02 18:25:03.606754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606740    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0efe75f2-184b-4ff5-bbf2-7791c96d9417" containerName="payload-pre-processing"
Nov 02 18:25:03.606831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606755    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efe75f2-184b-4ff5-bbf2-7791c96d9417" containerName="payload-pre-processing"
Nov 02 18:25:03.606831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606779    4476 cpu_manager.go:414] "containerMap: removing container" podUID="bc997153-2a84-43eb-9d52-3c2f00205680" containerName="payload-processing"
Nov 02 18:25:03.606831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606784    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc997153-2a84-43eb-9d52-3c2f00205680" containerName="payload-processing"
Nov 02 18:25:03.606831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606806    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" containerName="payload-pre-processing"
Nov 02 18:25:03.606831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606811    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" containerName="payload-pre-processing"
Nov 02 18:25:03.606993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606886    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc997153-2a84-43eb-9d52-3c2f00205680" containerName="payload-processing"
Nov 02 18:25:03.606993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606898    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8cd8953-99d2-44d4-9a0c-e4cf14e1d413" containerName="payload-pre-processing"
Nov 02 18:25:03.606993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.606906    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efe75f2-184b-4ff5-bbf2-7791c96d9417" containerName="payload-pre-processing"
Nov 02 18:25:03.611538 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.611518    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.655090 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.655060    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"]
Nov 02 18:25:03.658952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.658935    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.705758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.705726    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"]
Nov 02 18:25:03.725813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.725783    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.725946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.725825    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.725983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.725957    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726033 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.725994    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726033 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726024    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bns5\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726129    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvhk\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726162    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726266 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726206    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726304    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726393    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726429    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726551    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726580    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726606    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.726743 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726728    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726752    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.726786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.726775    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.761476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.761445    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"]
Nov 02 18:25:03.828052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828019    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828055    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828075    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828203    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828243    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828267    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828355    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828386    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828423    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828462    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828486    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828525    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828556    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828582    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828609    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bns5\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828697    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828712    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvhk\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828744    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828762    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828770    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828851    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.828926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828888    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.829740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.828984    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.829740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.829564    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.829740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.829683    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.829920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.829822    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.829920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.829838    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.830038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.829973    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.830865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.830831    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.831106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.831086    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.831192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.831172    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.831414 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.831400    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.885868 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.885835    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.886015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.885995    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.886135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.886118    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bns5\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:03.886707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.886681    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvhk\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.925196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.925172    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:03.974730 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:03.974697    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:04.368664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.368611    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"]
Nov 02 18:25:04.402659 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:25:04.402601    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6b9573_0e7e_448b_ad22_28bdf3ca61b9.slice/crio-e8d6631944b6af531974e292924e74e815187c60ca38c09e684b95848219a538 WatchSource:0}: Error finding container e8d6631944b6af531974e292924e74e815187c60ca38c09e684b95848219a538: Status 404 returned error can't find the container with id e8d6631944b6af531974e292924e74e815187c60ca38c09e684b95848219a538
Nov 02 18:25:04.405332 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.405301    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:25:04.405402 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.405370    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:25:04.405444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.405403    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:25:04.416974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.416946    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"]
Nov 02 18:25:04.417591 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:25:04.417572    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8056dae_a697_48e9_b811_1f60ebd61121.slice/crio-0f5054e4aa35faa0273503c85c27c911c39c1892de70804d2cdbc68485e18284 WatchSource:0}: Error finding container 0f5054e4aa35faa0273503c85c27c911c39c1892de70804d2cdbc68485e18284: Status 404 returned error can't find the container with id 0f5054e4aa35faa0273503c85c27c911c39c1892de70804d2cdbc68485e18284
Nov 02 18:25:04.420895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.420865    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:25:04.420999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.420937    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:25:04.420999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.420984    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:25:04.503421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.503387    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" event={"ID":"f8056dae-a697-48e9-b811-1f60ebd61121","Type":"ContainerStarted","Data":"cf0d9cb21bbe62daf899b048e3bf347ff0145c11dcc35091eeacd43568686d39"}
Nov 02 18:25:04.503598 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.503443    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" event={"ID":"f8056dae-a697-48e9-b811-1f60ebd61121","Type":"ContainerStarted","Data":"0f5054e4aa35faa0273503c85c27c911c39c1892de70804d2cdbc68485e18284"}
Nov 02 18:25:04.505781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.505743    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" event={"ID":"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9","Type":"ContainerStarted","Data":"44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309"}
Nov 02 18:25:04.505888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.505811    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" event={"ID":"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9","Type":"ContainerStarted","Data":"e8d6631944b6af531974e292924e74e815187c60ca38c09e684b95848219a538"}
Nov 02 18:25:04.623227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.623133    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podStartSLOduration=1.623119918 podStartE2EDuration="1.623119918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:04.619231344 +0000 UTC m=+3735.459837279" watchObservedRunningTime="2026-11-02 18:25:04.623119918 +0000 UTC m=+3735.463725919"
Nov 02 18:25:04.744653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.743675    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podStartSLOduration=1.743659605 podStartE2EDuration="1.743659605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:04.737525125 +0000 UTC m=+3735.578131058" watchObservedRunningTime="2026-11-02 18:25:04.743659605 +0000 UTC m=+3735.584265542"
Nov 02 18:25:04.926018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.925935    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:04.975793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:04.975744    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:05.412660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:05.412283    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:05.927936 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:05.927905    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:25:05.928353 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:05.927967    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 02 18:25:05.980656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:05.980045    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:25:05.980656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:05.980127    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 02 18:25:06.926032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:06.925983    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:25:06.926285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:06.926064    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 02 18:25:06.980654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:06.979690    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:25:06.980654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:06.979751    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 02 18:25:07.926092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:07.926056    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:25:07.926277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:07.926104    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 02 18:25:07.978653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:07.976818    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:25:07.978653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:07.976897    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 02 18:25:08.926158 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:08.926123    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:25:08.926567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:08.926173    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 02 18:25:08.977653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:08.976912    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:25:08.977653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:08.976970    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 02 18:25:09.925723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:09.925687    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 02 18:25:09.925933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:09.925757    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 02 18:25:09.988929 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:09.988777    4476 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 02 18:25:09.988929 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:09.988836    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 02 18:25:10.698988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.698949    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:10.931055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.931022    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:10.931320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.931298    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:10.932431 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.932410    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:25:10.981273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.981194    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:10.981607 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.981590    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:10.982303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:10.982282    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:25:15.144830 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:15.144795    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:16.976507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:16.976467    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:22.005904 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:22.005512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:25.598168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:25.598136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:25.829793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:25.829751    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:26.807488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:26.807449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:27.246798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:27.246767    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:27.840320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:27.840282    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:29.058494 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:25:29.058457    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:25:31.444411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:31.444380    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:31.672917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:31.672877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:32.204352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:32.204316    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:35.457669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.456581    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"]
Nov 02 18:25:35.460895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.460868    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.516860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.516830    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"]
Nov 02 18:25:35.561674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.561610    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:25:35.562076 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.562034    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:25:35.591914 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.591878    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.592140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.591921    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrs4\" (UniqueName: \"kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.592140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.592054    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.594883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.594825    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:35.694005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.693970    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrs4\" (UniqueName: \"kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.694836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.694808    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.695280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.695247    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.698079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.698055    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.709131 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.709056    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.751047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.751015    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrs4\" (UniqueName: \"kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4\") pod \"maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:35.774670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:35.774618    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:36.154694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:36.154670    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"]
Nov 02 18:25:36.156852 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:25:36.156815    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8bae08_4d33_405e_9da8_f00f91fc66ae.slice/crio-b786b517c69ee679c119b2d8c4f949b38c2b54ba6d0bc2fd76940f9f8a5eb37f WatchSource:0}: Error finding container b786b517c69ee679c119b2d8c4f949b38c2b54ba6d0bc2fd76940f9f8a5eb37f: Status 404 returned error can't find the container with id b786b517c69ee679c119b2d8c4f949b38c2b54ba6d0bc2fd76940f9f8a5eb37f
Nov 02 18:25:36.741413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:36.741332    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p" event={"ID":"fd8bae08-4d33-405e-9da8-f00f91fc66ae","Type":"ContainerStarted","Data":"48dea6fc9056ecfd05b28b758c92d12a1429dec500fe205d67700f15e02b19da"}
Nov 02 18:25:36.741413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:36.741397    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p" event={"ID":"fd8bae08-4d33-405e-9da8-f00f91fc66ae","Type":"ContainerStarted","Data":"b786b517c69ee679c119b2d8c4f949b38c2b54ba6d0bc2fd76940f9f8a5eb37f"}
Nov 02 18:25:36.858574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:36.858522    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p" podStartSLOduration=1.8585089099999998 podStartE2EDuration="1.85850891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:36.856185515 +0000 UTC m=+3767.696791450" watchObservedRunningTime="2026-11-02 18:25:36.85850891 +0000 UTC m=+3767.699114842"
Nov 02 18:25:37.301551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:37.301499    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:37.750999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:37.750965    4476 generic.go:356] "Generic (PLEG): container finished" podID="fd8bae08-4d33-405e-9da8-f00f91fc66ae" containerID="48dea6fc9056ecfd05b28b758c92d12a1429dec500fe205d67700f15e02b19da" exitCode=0
Nov 02 18:25:37.751404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:37.751051    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p" event={"ID":"fd8bae08-4d33-405e-9da8-f00f91fc66ae","Type":"ContainerDied","Data":"48dea6fc9056ecfd05b28b758c92d12a1429dec500fe205d67700f15e02b19da"}
Nov 02 18:25:38.919195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:38.919172    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:39.035913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.035812    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4\" (UniqueName: \"kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4\") pod \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") "
Nov 02 18:25:39.036235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.035964    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca\") pod \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") "
Nov 02 18:25:39.036235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.036085    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca\") pod \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\" (UID: \"fd8bae08-4d33-405e-9da8-f00f91fc66ae\") "
Nov 02 18:25:39.036899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.036869    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca" pod "fd8bae08-4d33-405e-9da8-f00f91fc66ae" (UID: "fd8bae08-4d33-405e-9da8-f00f91fc66ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:39.039227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.039200    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4" pod "fd8bae08-4d33-405e-9da8-f00f91fc66ae" (UID: "fd8bae08-4d33-405e-9da8-f00f91fc66ae"). InnerVolumeSpecName "kube-api-access-xnrs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:39.039872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.039845    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca" pod "fd8bae08-4d33-405e-9da8-f00f91fc66ae" (UID: "fd8bae08-4d33-405e-9da8-f00f91fc66ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:39.138371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.138327    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnrs4\" (UniqueName: \"kubernetes.io/projected/fd8bae08-4d33-405e-9da8-f00f91fc66ae-kube-api-access-xnrs4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:39.138371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.138370    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:39.138660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.138385    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd8bae08-4d33-405e-9da8-f00f91fc66ae-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:39.767701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.767578    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"
Nov 02 18:25:39.767701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.767614    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p" event={"ID":"fd8bae08-4d33-405e-9da8-f00f91fc66ae","Type":"ContainerDied","Data":"b786b517c69ee679c119b2d8c4f949b38c2b54ba6d0bc2fd76940f9f8a5eb37f"}
Nov 02 18:25:39.767701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:39.767691    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b786b517c69ee679c119b2d8c4f949b38c2b54ba6d0bc2fd76940f9f8a5eb37f"
Nov 02 18:25:40.106083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.105999    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"]
Nov 02 18:25:40.106577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.106561    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fd8bae08-4d33-405e-9da8-f00f91fc66ae" containerName="revoke-keys"
Nov 02 18:25:40.106577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.106577    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8bae08-4d33-405e-9da8-f00f91fc66ae" containerName="revoke-keys"
Nov 02 18:25:40.106721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.106688    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8bae08-4d33-405e-9da8-f00f91fc66ae" containerName="revoke-keys"
Nov 02 18:25:40.110206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.110186    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.167086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.167052    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"]
Nov 02 18:25:40.250489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.250455    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.250489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.250491    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcslf\" (UniqueName: \"kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.250740 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.250707    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.251562 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.251546    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:25:40.251702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.251684    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:25:40.352392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.352358    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.352564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.352460    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.352564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.352484    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcslf\" (UniqueName: \"kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.353302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.353223    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.355040 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.355016    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.409325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.409291    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcslf\" (UniqueName: \"kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf\") pod \"maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.425847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.425826    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:40.811892 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:40.811861    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"]
Nov 02 18:25:40.813243 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:25:40.813213    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee47b5aa_c9e8_4509_8895_e03f620cf4fe.slice/crio-dd19759d1b40d3e9d9d8255a8f8b2438ae3deacecf82d5665a14b907c2a288c3 WatchSource:0}: Error finding container dd19759d1b40d3e9d9d8255a8f8b2438ae3deacecf82d5665a14b907c2a288c3: Status 404 returned error can't find the container with id dd19759d1b40d3e9d9d8255a8f8b2438ae3deacecf82d5665a14b907c2a288c3
Nov 02 18:25:41.785355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:41.785323    4476 generic.go:356] "Generic (PLEG): container finished" podID="ee47b5aa-c9e8-4509-8895-e03f620cf4fe" containerID="de2f077a63f9c8f97b5bbeaf4a5391cfe9e92cfdadabf5dd03d3a03d10324d35" exitCode=0
Nov 02 18:25:41.785790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:41.785399    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd" event={"ID":"ee47b5aa-c9e8-4509-8895-e03f620cf4fe","Type":"ContainerDied","Data":"de2f077a63f9c8f97b5bbeaf4a5391cfe9e92cfdadabf5dd03d3a03d10324d35"}
Nov 02 18:25:41.785790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:41.785451    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd" event={"ID":"ee47b5aa-c9e8-4509-8895-e03f620cf4fe","Type":"ContainerStarted","Data":"dd19759d1b40d3e9d9d8255a8f8b2438ae3deacecf82d5665a14b907c2a288c3"}
Nov 02 18:25:42.260911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:42.260870    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:42.935294 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:42.935272    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:43.086034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.085956    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca\") pod \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") "
Nov 02 18:25:43.086247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.086225    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca\") pod \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") "
Nov 02 18:25:43.086359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.086342    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf\" (UniqueName: \"kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf\") pod \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\" (UID: \"ee47b5aa-c9e8-4509-8895-e03f620cf4fe\") "
Nov 02 18:25:43.086830 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.086791    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca" pod "ee47b5aa-c9e8-4509-8895-e03f620cf4fe" (UID: "ee47b5aa-c9e8-4509-8895-e03f620cf4fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:43.088804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.088769    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca" pod "ee47b5aa-c9e8-4509-8895-e03f620cf4fe" (UID: "ee47b5aa-c9e8-4509-8895-e03f620cf4fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:43.089333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.089306    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf" pod "ee47b5aa-c9e8-4509-8895-e03f620cf4fe" (UID: "ee47b5aa-c9e8-4509-8895-e03f620cf4fe"). InnerVolumeSpecName "kube-api-access-rcslf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:43.187851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.187816    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcslf\" (UniqueName: \"kubernetes.io/projected/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-kube-api-access-rcslf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:43.187851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.187847    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:43.188080 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.187860    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee47b5aa-c9e8-4509-8895-e03f620cf4fe-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:43.802104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.802022    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"
Nov 02 18:25:43.802104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.802052    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd" event={"ID":"ee47b5aa-c9e8-4509-8895-e03f620cf4fe","Type":"ContainerDied","Data":"dd19759d1b40d3e9d9d8255a8f8b2438ae3deacecf82d5665a14b907c2a288c3"}
Nov 02 18:25:43.802104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:43.802095    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd19759d1b40d3e9d9d8255a8f8b2438ae3deacecf82d5665a14b907c2a288c3"
Nov 02 18:25:46.968553 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:46.968514    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:50.991330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:50.991296    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"]
Nov 02 18:25:50.991835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:50.991553    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" podUID="f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" containerName="maas-api" containerID="cri-o://e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59" gracePeriod=30
Nov 02 18:25:51.040982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.040952    4476 scope.go:122] "RemoveContainer" containerID="b5de08975499f7871a96155f9cfb002bdcac1c4b0abc2e57b2b4c846d96fd911"
Nov 02 18:25:51.170149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.170105    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:51.295944 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.295911    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:25:51.398654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.393994    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca\") pod \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") "
Nov 02 18:25:51.398654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.394095    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\") pod \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") "
Nov 02 18:25:51.398654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.394182    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs\") pod \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") "
Nov 02 18:25:51.398654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.394287    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv\" (UniqueName: \"kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv\") pod \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\" (UID: \"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9\") "
Nov 02 18:25:51.398654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.395605    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca" pod "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" (UID: "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:51.421653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.418199    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls" pod "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" (UID: "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:51.421653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.418340    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv" pod "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" (UID: "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9"). InnerVolumeSpecName "kube-api-access-5d4tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:51.433376 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.432192    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs" pod "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" (UID: "f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:51.496254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.496221    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4tv\" (UniqueName: \"kubernetes.io/projected/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-kube-api-access-5d4tv\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:51.496254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.496251    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:51.496254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.496261    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:51.496511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.496270    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:25:51.869207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.869159    4476 generic.go:356] "Generic (PLEG): container finished" podID="f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" containerID="e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59" exitCode=0
Nov 02 18:25:51.869423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.869227    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"
Nov 02 18:25:51.869423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.869369    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" event={"ID":"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9","Type":"ContainerDied","Data":"e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59"}
Nov 02 18:25:51.869539 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.869425    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz" event={"ID":"f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9","Type":"ContainerDied","Data":"1d783d29cbc0f270c0991fa0c961aac6da4774bbd49346ec28107927dee4787f"}
Nov 02 18:25:51.869539 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.869456    4476 scope.go:122] "RemoveContainer" containerID="e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59"
Nov 02 18:25:51.883041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.883023    4476 scope.go:122] "RemoveContainer" containerID="e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59"
Nov 02 18:25:51.883370 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:25:51.883345    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59\": container with ID starting with e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59 not found: ID does not exist" containerID="e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59"
Nov 02 18:25:51.883439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:51.883380    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59"} err="failed to get container status \"e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59\": rpc error: code = NotFound desc = could not find container \"e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59\": container with ID starting with e960b8193726800b5275615ba791a2457e306956c7a2365bf9e1c3cfe8a80c59 not found: ID does not exist"
Nov 02 18:25:52.034378 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:52.034343    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"]
Nov 02 18:25:52.037915 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:52.037891    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-508b8f-74888cf67d-hlzbz"]
Nov 02 18:25:52.868101 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:52.868071    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:53.078522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:53.078485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:53.119742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:53.119671    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"]
Nov 02 18:25:53.147505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:53.147477    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-508b8f-pv75p"]
Nov 02 18:25:53.379985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:53.379855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:53.384325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:53.384297    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" path="/var/lib/kubelet/pods/f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9/volumes"
Nov 02 18:25:53.384768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:53.384753    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8bae08-4d33-405e-9da8-f00f91fc66ae" path="/var/lib/kubelet/pods/fd8bae08-4d33-405e-9da8-f00f91fc66ae/volumes"
Nov 02 18:25:56.366203 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:56.366168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:56.575883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:56.575208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:58.214494 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:25:58.214454    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:59.058550 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:25:59.058510    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:26:01.362504 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:01.362463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:03.111450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:03.111409    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:07.835682 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:07.835646    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:08.642278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.642242    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"]
Nov 02 18:26:08.643086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.643063    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" containerName="maas-api"
Nov 02 18:26:08.643141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.643090    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" containerName="maas-api"
Nov 02 18:26:08.643141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.643122    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ee47b5aa-c9e8-4509-8895-e03f620cf4fe" containerName="revoke-keys"
Nov 02 18:26:08.643141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.643130    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee47b5aa-c9e8-4509-8895-e03f620cf4fe" containerName="revoke-keys"
Nov 02 18:26:08.643301 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.643287    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee47b5aa-c9e8-4509-8895-e03f620cf4fe" containerName="revoke-keys"
Nov 02 18:26:08.643337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.643317    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32bc4e6-d62f-4524-aaf0-0e3fda3b11c9" containerName="maas-api"
Nov 02 18:26:08.647382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.647362    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.702023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.701992    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"]
Nov 02 18:26:08.712887 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.712854    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.713036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.712941    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgf7\" (UniqueName: \"kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.713036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.713006    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.746310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.746277    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:26:08.746310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.746300    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:26:08.815083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.815050    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.815299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.815111    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgf7\" (UniqueName: \"kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.815299 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.815164    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.816067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.816039    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.817996 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.817965    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.872571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.872543    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgf7\" (UniqueName: \"kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7\") pod \"maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:08.962431 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:08.962347    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:09.351371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:09.351345    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"]
Nov 02 18:26:09.353467 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:26:09.353434    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2698860_cdd0_42dd_bf71_356cc6072414.slice/crio-0e7db80a43cea5a2c521a0417c781653016373fd0d47c494ff4535306f6b5b2d WatchSource:0}: Error finding container 0e7db80a43cea5a2c521a0417c781653016373fd0d47c494ff4535306f6b5b2d: Status 404 returned error can't find the container with id 0e7db80a43cea5a2c521a0417c781653016373fd0d47c494ff4535306f6b5b2d
Nov 02 18:26:10.014250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.014154    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82" event={"ID":"b2698860-cdd0-42dd-bf71-356cc6072414","Type":"ContainerStarted","Data":"938d242ec5a5b65e6747358239f1a77ae21be4ed9f3706252a9fa49438782fdd"}
Nov 02 18:26:10.014250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.014220    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82" event={"ID":"b2698860-cdd0-42dd-bf71-356cc6072414","Type":"ContainerStarted","Data":"0e7db80a43cea5a2c521a0417c781653016373fd0d47c494ff4535306f6b5b2d"}
Nov 02 18:26:10.052134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.052091    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:10.132232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.132175    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82" podStartSLOduration=2.132160228 podStartE2EDuration="2.132160228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:10.128249621 +0000 UTC m=+3800.968855556" watchObservedRunningTime="2026-11-02 18:26:10.132160228 +0000 UTC m=+3800.972766160"
Nov 02 18:26:10.371179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.371147    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"]
Nov 02 18:26:10.375280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.375255    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.428316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.428287    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\""
Nov 02 18:26:10.432394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.432368    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"]
Nov 02 18:26:10.434759 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.434740    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.434813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.434771    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.434813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.434799    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.434912 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.434901    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.434955 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.434944    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnnj\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.435010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.434967    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.435091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.435074    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.435142 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.435114    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.435234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.435211    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.535876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.535847    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.535898    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.535926    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536028    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536171    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536271    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnnj\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536342 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536311    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536357    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536410    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.536464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536409    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.537019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.536988    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.537145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.537024    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.537145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.537054    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.537695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.537646    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.539685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.539611    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.540689 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.540663    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.593324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.593293    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.593489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.593330    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnnj\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-gbzt9\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:10.690228 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:10.690151    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:11.023501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:11.023400    4476 generic.go:356] "Generic (PLEG): container finished" podID="b2698860-cdd0-42dd-bf71-356cc6072414" containerID="938d242ec5a5b65e6747358239f1a77ae21be4ed9f3706252a9fa49438782fdd" exitCode=0
Nov 02 18:26:11.023501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:11.023493    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82" event={"ID":"b2698860-cdd0-42dd-bf71-356cc6072414","Type":"ContainerDied","Data":"938d242ec5a5b65e6747358239f1a77ae21be4ed9f3706252a9fa49438782fdd"}
Nov 02 18:26:11.135318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:11.135292    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"]
Nov 02 18:26:11.136677 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:26:11.136617    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9e95d7_9f83_4388_b59b_7e14307e88ba.slice/crio-1a2fd112f7033b99c9c797c370d5fe7a5f112107c62935c4d017b442fc1d3573 WatchSource:0}: Error finding container 1a2fd112f7033b99c9c797c370d5fe7a5f112107c62935c4d017b442fc1d3573: Status 404 returned error can't find the container with id 1a2fd112f7033b99c9c797c370d5fe7a5f112107c62935c4d017b442fc1d3573
Nov 02 18:26:11.139594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:11.139554    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:26:11.139747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:11.139659    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:26:11.139747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:11.139692    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:26:12.034120 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.034084    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" event={"ID":"af9e95d7-9f83-4388-b59b-7e14307e88ba","Type":"ContainerStarted","Data":"9705eaa320085d8fe305beb20f9205e88fe8cd64ed0a9d9c13b667a162cf3af4"}
Nov 02 18:26:12.034548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.034149    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" event={"ID":"af9e95d7-9f83-4388-b59b-7e14307e88ba","Type":"ContainerStarted","Data":"1a2fd112f7033b99c9c797c370d5fe7a5f112107c62935c4d017b442fc1d3573"}
Nov 02 18:26:12.173654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.172665    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podStartSLOduration=2.172642941 podStartE2EDuration="2.172642941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:12.154936994 +0000 UTC m=+3802.995542922" watchObservedRunningTime="2026-11-02 18:26:12.172642941 +0000 UTC m=+3803.013248870"
Nov 02 18:26:12.275749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.275723    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:12.374567 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.374537    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca\") pod \"b2698860-cdd0-42dd-bf71-356cc6072414\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") "
Nov 02 18:26:12.374785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.374661    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca\") pod \"b2698860-cdd0-42dd-bf71-356cc6072414\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") "
Nov 02 18:26:12.374785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.374682    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7\" (UniqueName: \"kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7\") pod \"b2698860-cdd0-42dd-bf71-356cc6072414\" (UID: \"b2698860-cdd0-42dd-bf71-356cc6072414\") "
Nov 02 18:26:12.374966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.374943    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca" pod "b2698860-cdd0-42dd-bf71-356cc6072414" (UID: "b2698860-cdd0-42dd-bf71-356cc6072414"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:12.375153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.375133    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:12.377052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.377025    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7" pod "b2698860-cdd0-42dd-bf71-356cc6072414" (UID: "b2698860-cdd0-42dd-bf71-356cc6072414"). InnerVolumeSpecName "kube-api-access-lmgf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:12.377159 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.377044    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca" pod "b2698860-cdd0-42dd-bf71-356cc6072414" (UID: "b2698860-cdd0-42dd-bf71-356cc6072414"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:26:12.476663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.476609    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2698860-cdd0-42dd-bf71-356cc6072414-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:12.476663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.476665    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmgf7\" (UniqueName: \"kubernetes.io/projected/b2698860-cdd0-42dd-bf71-356cc6072414-kube-api-access-lmgf7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:12.690352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.690265    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:12.692051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.692026    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:12.692175 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.692076    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:12.806744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:12.806702    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:13.067465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:13.067378    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"
Nov 02 18:26:13.067465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:13.067403    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82" event={"ID":"b2698860-cdd0-42dd-bf71-356cc6072414","Type":"ContainerDied","Data":"0e7db80a43cea5a2c521a0417c781653016373fd0d47c494ff4535306f6b5b2d"}
Nov 02 18:26:13.067923 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:13.067484    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e7db80a43cea5a2c521a0417c781653016373fd0d47c494ff4535306f6b5b2d"
Nov 02 18:26:13.690728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:13.690697    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:13.690915 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:13.690759    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:14.690773 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:14.690737    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:14.691187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:14.690792    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:15.690846 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:15.690809    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:15.691267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:15.690863    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:16.690786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:16.690746    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:16.690973 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:16.690811    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:17.691380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.691333    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:17.691953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.691397    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:17.736951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.736915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:17.748544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.748506    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"]
Nov 02 18:26:17.750030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.750003    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b2698860-cdd0-42dd-bf71-356cc6072414" containerName="revoke-keys"
Nov 02 18:26:17.750184 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.750171    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2698860-cdd0-42dd-bf71-356cc6072414" containerName="revoke-keys"
Nov 02 18:26:17.750573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.750556    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2698860-cdd0-42dd-bf71-356cc6072414" containerName="revoke-keys"
Nov 02 18:26:17.755926 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.755896    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.817178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.814270    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"]
Nov 02 18:26:17.851608 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.851569    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2hf\" (UniqueName: \"kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.851974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.851947    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.852212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.852174    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.856895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.856872    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:26:17.857756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.857710    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:26:17.986465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.986370    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2hf\" (UniqueName: \"kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.986780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.986752    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.986970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.986948    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:17.994580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:17.991851    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:18.013978 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:18.013952    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:18.053394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:18.053353    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2hf\" (UniqueName: \"kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf\") pod \"maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:18.070874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:18.070838    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:18.569614 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:18.569583    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"]
Nov 02 18:26:18.572109 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:26:18.572075    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3a05a4_94a9_44b9_8b82_d7c8cc80ef3e.slice/crio-78aa99c88b9645177dc256040f35ee1ae0eb6e3d11a55b462f42c47a879386dd WatchSource:0}: Error finding container 78aa99c88b9645177dc256040f35ee1ae0eb6e3d11a55b462f42c47a879386dd: Status 404 returned error can't find the container with id 78aa99c88b9645177dc256040f35ee1ae0eb6e3d11a55b462f42c47a879386dd
Nov 02 18:26:18.691218 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:18.691175    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 02 18:26:18.691427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:18.691226    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 02 18:26:19.149684 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:19.149646    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk" event={"ID":"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e","Type":"ContainerStarted","Data":"661c2fd53f9b12e141df289a54b110fcba9d7ab6018945935d0d0eb5ce9dfa5b"}
Nov 02 18:26:19.149951 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:19.149700    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk" event={"ID":"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e","Type":"ContainerStarted","Data":"78aa99c88b9645177dc256040f35ee1ae0eb6e3d11a55b462f42c47a879386dd"}
Nov 02 18:26:19.265562 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:19.265462    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk" podStartSLOduration=2.265449362 podStartE2EDuration="2.265449362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:19.26297091 +0000 UTC m=+3810.103576843" watchObservedRunningTime="2026-11-02 18:26:19.265449362 +0000 UTC m=+3810.106055293"
Nov 02 18:26:20.149401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:20.149357    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:20.150682 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:20.150620    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:20.152106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:20.152079    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:26:20.160398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:20.160369    4476 generic.go:356] "Generic (PLEG): container finished" podID="fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" containerID="661c2fd53f9b12e141df289a54b110fcba9d7ab6018945935d0d0eb5ce9dfa5b" exitCode=0
Nov 02 18:26:20.160530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:20.160457    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk" event={"ID":"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e","Type":"ContainerDied","Data":"661c2fd53f9b12e141df289a54b110fcba9d7ab6018945935d0d0eb5ce9dfa5b"}
Nov 02 18:26:21.335572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.335549    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:21.457344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.457256    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf\" (UniqueName: \"kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf\") pod \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") "
Nov 02 18:26:21.457529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.457345    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca\") pod \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") "
Nov 02 18:26:21.457529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.457486    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca\") pod \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\" (UID: \"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e\") "
Nov 02 18:26:21.457938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.457906    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca" pod "fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" (UID: "fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:21.460047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.460012    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca" pod "fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" (UID: "fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:26:21.460191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.460122    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf" pod "fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" (UID: "fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e"). InnerVolumeSpecName "kube-api-access-jl2hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:21.559392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.559360    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2hf\" (UniqueName: \"kubernetes.io/projected/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-kube-api-access-jl2hf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:21.559392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.559388    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:21.559392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.559399    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:21.932404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.932367    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"]
Nov 02 18:26:21.933000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.932986    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" containerName="revoke-keys"
Nov 02 18:26:21.933055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.933001    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" containerName="revoke-keys"
Nov 02 18:26:21.933104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.933097    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" containerName="revoke-keys"
Nov 02 18:26:21.936495 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.936480    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:21.994533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:21.994501    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"]
Nov 02 18:26:22.066297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.066262    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.066485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.066325    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5vv\" (UniqueName: \"kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.066545 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.066518    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.167530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.167488    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.167756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.167681    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.167833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.167757    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5vv\" (UniqueName: \"kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.168443 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.168417    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.170773 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.170745    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.179676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.179655    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"
Nov 02 18:26:22.179802 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.179686    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk" event={"ID":"fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e","Type":"ContainerDied","Data":"78aa99c88b9645177dc256040f35ee1ae0eb6e3d11a55b462f42c47a879386dd"}
Nov 02 18:26:22.179802 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.179746    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78aa99c88b9645177dc256040f35ee1ae0eb6e3d11a55b462f42c47a879386dd"
Nov 02 18:26:22.228994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.228972    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5vv\" (UniqueName: \"kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv\") pod \"maas-api-revoke-sub-6b794d09-859h7\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.249901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.249877    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:22.636098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.636071    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"]
Nov 02 18:26:22.636940 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:26:22.636916    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f93889_423f_47a0_a1ec_782e3450be57.slice/crio-6580dc3dad59d321cbf504e01c4e13510d5e6d82d409fa65dea5c2b9d2b275b9 WatchSource:0}: Error finding container 6580dc3dad59d321cbf504e01c4e13510d5e6d82d409fa65dea5c2b9d2b275b9: Status 404 returned error can't find the container with id 6580dc3dad59d321cbf504e01c4e13510d5e6d82d409fa65dea5c2b9d2b275b9
Nov 02 18:26:22.744295 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.744262    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:22.949712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:22.949689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:23.190406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:23.190373    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7" event={"ID":"31f93889-423f-47a0-a1ec-782e3450be57","Type":"ContainerStarted","Data":"23afb7a62eb3c660a830c0ba192e4f08579ef38b9ad78e54ea31f5d73864fc7b"}
Nov 02 18:26:23.190587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:23.190430    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7" event={"ID":"31f93889-423f-47a0-a1ec-782e3450be57","Type":"ContainerStarted","Data":"6580dc3dad59d321cbf504e01c4e13510d5e6d82d409fa65dea5c2b9d2b275b9"}
Nov 02 18:26:23.307738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:23.307607    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7" podStartSLOduration=2.307593636 podStartE2EDuration="2.307593636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:23.307061656 +0000 UTC m=+3814.147667589" watchObservedRunningTime="2026-11-02 18:26:23.307593636 +0000 UTC m=+3814.148199567"
Nov 02 18:26:24.198584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:24.198551    4476 generic.go:356] "Generic (PLEG): container finished" podID="31f93889-423f-47a0-a1ec-782e3450be57" containerID="23afb7a62eb3c660a830c0ba192e4f08579ef38b9ad78e54ea31f5d73864fc7b" exitCode=0
Nov 02 18:26:24.199042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:24.198654    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7" event={"ID":"31f93889-423f-47a0-a1ec-782e3450be57","Type":"ContainerDied","Data":"23afb7a62eb3c660a830c0ba192e4f08579ef38b9ad78e54ea31f5d73864fc7b"}
Nov 02 18:26:25.353997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.353971    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:25.405364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.405338    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca\") pod \"31f93889-423f-47a0-a1ec-782e3450be57\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") "
Nov 02 18:26:25.405502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.405483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv\" (UniqueName: \"kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv\") pod \"31f93889-423f-47a0-a1ec-782e3450be57\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") "
Nov 02 18:26:25.405577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.405562    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca\") pod \"31f93889-423f-47a0-a1ec-782e3450be57\" (UID: \"31f93889-423f-47a0-a1ec-782e3450be57\") "
Nov 02 18:26:25.405906 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.405874    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca" pod "31f93889-423f-47a0-a1ec-782e3450be57" (UID: "31f93889-423f-47a0-a1ec-782e3450be57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:25.406237 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.406051    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:25.407613 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.407589    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca" pod "31f93889-423f-47a0-a1ec-782e3450be57" (UID: "31f93889-423f-47a0-a1ec-782e3450be57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:26:25.407758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.407740    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv" pod "31f93889-423f-47a0-a1ec-782e3450be57" (UID: "31f93889-423f-47a0-a1ec-782e3450be57"). InnerVolumeSpecName "kube-api-access-vs5vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:25.507669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.507559    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs5vv\" (UniqueName: \"kubernetes.io/projected/31f93889-423f-47a0-a1ec-782e3450be57-kube-api-access-vs5vv\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:25.507669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:25.507593    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f93889-423f-47a0-a1ec-782e3450be57-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:26:26.213326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:26.213289    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7" event={"ID":"31f93889-423f-47a0-a1ec-782e3450be57","Type":"ContainerDied","Data":"6580dc3dad59d321cbf504e01c4e13510d5e6d82d409fa65dea5c2b9d2b275b9"}
Nov 02 18:26:26.213326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:26.213308    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"
Nov 02 18:26:26.213593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:26.213358    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6580dc3dad59d321cbf504e01c4e13510d5e6d82d409fa65dea5c2b9d2b275b9"
Nov 02 18:26:27.391042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:27.390999    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:29.060472 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:26:29.060434    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:26:32.210447 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:32.210361    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:34.364315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:34.364276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:34.573271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:34.573239    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:36.872849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:36.872814    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:37.055783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:37.055744    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:37.262407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:37.262296    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:41.223000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:41.222961    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:41.417752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:41.417694    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:41.989728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:41.989692    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:43.018712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:43.018685    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:43.235865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:43.235833    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:46.790369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:46.790331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:49.312646 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:49.312547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:49.521650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:49.519305    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:51.674422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:51.674389    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:55.759246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:55.759208    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:55.947400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:55.947363    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:56.692268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:56.692230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:58.032280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.032242    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:58.319738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.314838    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:58.751366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.751330    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"]
Nov 02 18:26:58.751970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.751954    4476 cpu_manager.go:414] "containerMap: removing container" podUID="31f93889-423f-47a0-a1ec-782e3450be57" containerName="revoke-keys"
Nov 02 18:26:58.752039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.751971    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f93889-423f-47a0-a1ec-782e3450be57" containerName="revoke-keys"
Nov 02 18:26:58.752087 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.752078    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f93889-423f-47a0-a1ec-782e3450be57" containerName="revoke-keys"
Nov 02 18:26:58.755643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.755608    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.808294 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.808254    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4bef29666866\""
Nov 02 18:26:58.808461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.808295    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n74nw\""
Nov 02 18:26:58.844207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.844158    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzp6\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.844375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.844298    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.844375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.844362    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.844544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.844526    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.858559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.857863    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 18:26:58.858559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.857890    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 18:26:58.858559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.857982    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 18:26:58.858559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.858004    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 18:26:58.864989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.864959    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"]
Nov 02 18:26:58.946133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.946099    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzp6\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.946318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.946150    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.946318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.946289    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.946444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.946425    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.947058 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.947034    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.947525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.947499    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:58.949001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:58.948979    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:59.003750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:59.003664    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzp6\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6\") pod \"payload-processing-external-model-596fb6c5c4-vtvw7\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:59.061162 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:26:59.061119    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:26:59.071371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:59.071346    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:26:59.511503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:26:59.511466    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"]
Nov 02 18:26:59.522746 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:26:59.522716    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ba99a8_c7fd_49b3_8095_d11da66ca393.slice/crio-b7b13a10aa182c72d2d056b6ed090cc5975586efddbbe643ec98b1147ebc0e6a WatchSource:0}: Error finding container b7b13a10aa182c72d2d056b6ed090cc5975586efddbbe643ec98b1147ebc0e6a: Status 404 returned error can't find the container with id b7b13a10aa182c72d2d056b6ed090cc5975586efddbbe643ec98b1147ebc0e6a
Nov 02 18:27:00.507548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:00.507497    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7" event={"ID":"38ba99a8-c7fd-49b3-8095-d11da66ca393","Type":"ContainerStarted","Data":"b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8"}
Nov 02 18:27:00.508107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:00.507566    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7" event={"ID":"38ba99a8-c7fd-49b3-8095-d11da66ca393","Type":"ContainerStarted","Data":"b7b13a10aa182c72d2d056b6ed090cc5975586efddbbe643ec98b1147ebc0e6a"}
Nov 02 18:27:00.508107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:00.507713    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:27:00.626453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:00.626400    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7" podStartSLOduration=2.626387963 podStartE2EDuration="2.626387963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:00.625839296 +0000 UTC m=+3851.466445232" watchObservedRunningTime="2026-11-02 18:27:00.626387963 +0000 UTC m=+3851.466993897"
Nov 02 18:27:01.330077 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:01.330028    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"]
Nov 02 18:27:01.695207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:01.695141    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:01.932182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:01.932151    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:04.542326 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:04.542259    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:04.729434 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:04.729386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:06.706751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:06.706711    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:07.523721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.523681    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7" podUID="38ba99a8-c7fd-49b3-8095-d11da66ca393" containerName="payload-processing" containerID="cri-o://b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8" gracePeriod=25
Nov 02 18:27:07.803353 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.803323    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:27:07.875827 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.875792    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials\") pod \"38ba99a8-c7fd-49b3-8095-d11da66ca393\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") "
Nov 02 18:27:07.876051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.875957    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume\") pod \"38ba99a8-c7fd-49b3-8095-d11da66ca393\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") "
Nov 02 18:27:07.876130 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.876072    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume\") pod \"38ba99a8-c7fd-49b3-8095-d11da66ca393\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") "
Nov 02 18:27:07.876248 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.876167    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6\") pod \"38ba99a8-c7fd-49b3-8095-d11da66ca393\" (UID: \"38ba99a8-c7fd-49b3-8095-d11da66ca393\") "
Nov 02 18:27:07.876386 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.876356    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume" pod "38ba99a8-c7fd-49b3-8095-d11da66ca393" (UID: "38ba99a8-c7fd-49b3-8095-d11da66ca393"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:07.876791 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.876760    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:07.876927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.876796    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume" pod "38ba99a8-c7fd-49b3-8095-d11da66ca393" (UID: "38ba99a8-c7fd-49b3-8095-d11da66ca393"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:07.878428 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.878405    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials" pod "38ba99a8-c7fd-49b3-8095-d11da66ca393" (UID: "38ba99a8-c7fd-49b3-8095-d11da66ca393"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:07.878654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.878616    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6" pod "38ba99a8-c7fd-49b3-8095-d11da66ca393" (UID: "38ba99a8-c7fd-49b3-8095-d11da66ca393"). InnerVolumeSpecName "kube-api-access-rhzp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:07.915505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.915477    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:07.978497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.978466    4476 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38ba99a8-c7fd-49b3-8095-d11da66ca393-routing-overlay-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:07.978497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.978491    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzp6\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-kube-api-access-rhzp6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:07.978497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:07.978501    4476 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38ba99a8-c7fd-49b3-8095-d11da66ca393-provider-credentials\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:08.368351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.368315    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"]
Nov 02 18:27:08.368964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.368947    4476 cpu_manager.go:414] "containerMap: removing container" podUID="38ba99a8-c7fd-49b3-8095-d11da66ca393" containerName="payload-processing"
Nov 02 18:27:08.369054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.368968    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ba99a8-c7fd-49b3-8095-d11da66ca393" containerName="payload-processing"
Nov 02 18:27:08.369097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.369092    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ba99a8-c7fd-49b3-8095-d11da66ca393" containerName="payload-processing"
Nov 02 18:27:08.373010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.372979    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.429804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.429019    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-662a80-openshift-default-dockercfg-qdcll\""
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485118    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485188    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485233    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485269    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485413    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485446    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485491    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485532    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzxm\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.485652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.485582    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.520132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.518272    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"]
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589144    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589202    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589247    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589290    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzxm\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589340    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589416    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589463    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589499    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.589538    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.590067    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.590859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.590333    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.601600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.600783    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.607741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.604040    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.607741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.604691    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.607741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.607237    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.609685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.609654    4476 generic.go:356] "Generic (PLEG): container finished" podID="38ba99a8-c7fd-49b3-8095-d11da66ca393" containerID="b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8" exitCode=0
Nov 02 18:27:08.610225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.610183    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"
Nov 02 18:27:08.610341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.610295    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7" event={"ID":"38ba99a8-c7fd-49b3-8095-d11da66ca393","Type":"ContainerDied","Data":"b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8"}
Nov 02 18:27:08.610415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.610391    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7" event={"ID":"38ba99a8-c7fd-49b3-8095-d11da66ca393","Type":"ContainerDied","Data":"b7b13a10aa182c72d2d056b6ed090cc5975586efddbbe643ec98b1147ebc0e6a"}
Nov 02 18:27:08.610477 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.610445    4476 scope.go:122] "RemoveContainer" containerID="b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8"
Nov 02 18:27:08.618120 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.618092    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.642303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.641981    4476 scope.go:122] "RemoveContainer" containerID="b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8"
Nov 02 18:27:08.642995 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:08.642677    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8\": container with ID starting with b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8 not found: ID does not exist" containerID="b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8"
Nov 02 18:27:08.642995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.642720    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8"} err="failed to get container status \"b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8\": rpc error: code = NotFound desc = could not find container \"b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8\": container with ID starting with b9fe8126389f438984e279b9b5d553d7639ce3470a741d3f06f73bab2d85e4c8 not found: ID does not exist"
Nov 02 18:27:08.667464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.667174    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.667464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.667387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzxm\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm\") pod \"e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") " pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.687882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.687843    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:08.755752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.754227    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:08.781084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.781050    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"]
Nov 02 18:27:08.788714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.788680    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-vtvw7"]
Nov 02 18:27:08.954513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:08.954411    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:09.155578 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:27:09.155545    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d55225d_9bad_48ad_a7d8_3f0a741ec985.slice/crio-89acce6f8c10649bedcef28c4d9f4d4673cb7705883e9abe62c33388589ccbb0 WatchSource:0}: Error finding container 89acce6f8c10649bedcef28c4d9f4d4673cb7705883e9abe62c33388589ccbb0: Status 404 returned error can't find the container with id 89acce6f8c10649bedcef28c4d9f4d4673cb7705883e9abe62c33388589ccbb0
Nov 02 18:27:09.155946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.155912    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"]
Nov 02 18:27:09.159128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.159096    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:09.159254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.159174    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:09.159254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.159221    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:09.386164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.386128    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ba99a8-c7fd-49b3-8095-d11da66ca393" path="/var/lib/kubelet/pods/38ba99a8-c7fd-49b3-8095-d11da66ca393/volumes"
Nov 02 18:27:09.621395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.621360    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" event={"ID":"7d55225d-9bad-48ad-a7d8-3f0a741ec985","Type":"ContainerStarted","Data":"426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"}
Nov 02 18:27:09.621589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.621433    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" event={"ID":"7d55225d-9bad-48ad-a7d8-3f0a741ec985","Type":"ContainerStarted","Data":"89acce6f8c10649bedcef28c4d9f4d4673cb7705883e9abe62c33388589ccbb0"}
Nov 02 18:27:09.688586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.688554    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:09.743183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:09.743108    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podStartSLOduration=1.7430899869999998 podStartE2EDuration="1.743089987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:09.738032097 +0000 UTC m=+3860.578638028" watchObservedRunningTime="2026-11-02 18:27:09.743089987 +0000 UTC m=+3860.583695922"
Nov 02 18:27:10.694701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:10.694669    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:10.926051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:10.926014    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:11.637589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:11.637563    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:11.638557 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:11.638536    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:27:12.827402 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:12.827370    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:12.956139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:12.956091    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"]
Nov 02 18:27:13.002785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:13.002754    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a8da6e-xhhkd"]
Nov 02 18:27:13.021641 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:13.021603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:13.299518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:13.299485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:13.395551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:13.395517    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee47b5aa-c9e8-4509-8895-e03f620cf4fe" path="/var/lib/kubelet/pods/ee47b5aa-c9e8-4509-8895-e03f620cf4fe/volumes"
Nov 02 18:27:13.503444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:13.503406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:13.721271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:13.721240    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:14.123255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:14.123225    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:16.634179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:16.634149    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:16.826099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:16.826066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:17.969013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:17.968984    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:18.170270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:18.170229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:18.396285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:18.396244    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:20.457501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:20.457466    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:21.558346 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:21.558319    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"]
Nov 02 18:27:21.895803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:21.895766    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:22.094903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.094847    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:22.281974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.281882    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"]
Nov 02 18:27:22.282211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.282179    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" podUID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" containerName="maas-api" containerID="cri-o://1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69" gracePeriod=30
Nov 02 18:27:22.314571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.314530    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" podUID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.199:8443/health\": dial tcp 10.133.0.199:8443: connect: connection refused"
Nov 02 18:27:22.433211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.432941    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:22.608084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.608051    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:27:22.740106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.740073    4476 generic.go:356] "Generic (PLEG): container finished" podID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" containerID="1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69" exitCode=0
Nov 02 18:27:22.740300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.740137    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"
Nov 02 18:27:22.740300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.740256    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" event={"ID":"9b55cd26-1b6d-42ba-925d-84f49ab4304e","Type":"ContainerDied","Data":"1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69"}
Nov 02 18:27:22.740426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.740309    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm" event={"ID":"9b55cd26-1b6d-42ba-925d-84f49ab4304e","Type":"ContainerDied","Data":"a606242e7fd3543c25bafc2948c1e67e45d3bc7ec40652fab659a9248ef8ea4c"}
Nov 02 18:27:22.740426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.740339    4476 scope.go:122] "RemoveContainer" containerID="1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69"
Nov 02 18:27:22.754249 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.754222    4476 scope.go:122] "RemoveContainer" containerID="1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69"
Nov 02 18:27:22.754605 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:22.754577    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69\": container with ID starting with 1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69 not found: ID does not exist" containerID="1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69"
Nov 02 18:27:22.754696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.754618    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69"} err="failed to get container status \"1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69\": rpc error: code = NotFound desc = could not find container \"1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69\": container with ID starting with 1f19113f664d8aefe6cdd254f440ee79da5526d519864d3809b92b23dc751a69 not found: ID does not exist"
Nov 02 18:27:22.774977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.774941    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca\") pod \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") "
Nov 02 18:27:22.775145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.775035    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s\" (UniqueName: \"kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s\") pod \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") "
Nov 02 18:27:22.775145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.775095    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\") pod \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") "
Nov 02 18:27:22.775145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.775141    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs\") pod \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\" (UID: \"9b55cd26-1b6d-42ba-925d-84f49ab4304e\") "
Nov 02 18:27:22.775375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.775346    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca" pod "9b55cd26-1b6d-42ba-925d-84f49ab4304e" (UID: "9b55cd26-1b6d-42ba-925d-84f49ab4304e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:22.775548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.775529    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b55cd26-1b6d-42ba-925d-84f49ab4304e-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:22.778285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.778259    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s" pod "9b55cd26-1b6d-42ba-925d-84f49ab4304e" (UID: "9b55cd26-1b6d-42ba-925d-84f49ab4304e"). InnerVolumeSpecName "kube-api-access-xlw5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:22.778285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.778259    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs" pod "9b55cd26-1b6d-42ba-925d-84f49ab4304e" (UID: "9b55cd26-1b6d-42ba-925d-84f49ab4304e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:22.780790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.780767    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls" pod "9b55cd26-1b6d-42ba-925d-84f49ab4304e" (UID: "9b55cd26-1b6d-42ba-925d-84f49ab4304e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:22.876938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.876908    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:22.876938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.876936    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b55cd26-1b6d-42ba-925d-84f49ab4304e-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:22.877124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:22.876947    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlw5s\" (UniqueName: \"kubernetes.io/projected/9b55cd26-1b6d-42ba-925d-84f49ab4304e-kube-api-access-xlw5s\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:23.211734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:23.195689    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"]
Nov 02 18:27:23.217593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:23.214931    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e4b90c-58d67549-8dgcm"]
Nov 02 18:27:23.392537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:23.392489    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" path="/var/lib/kubelet/pods/9b55cd26-1b6d-42ba-925d-84f49ab4304e/volumes"
Nov 02 18:27:24.415432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:24.415404    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:25.956594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:25.956557    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"]
Nov 02 18:27:25.957396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:25.957371    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" containerName="maas-api"
Nov 02 18:27:25.957511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:25.957398    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" containerName="maas-api"
Nov 02 18:27:25.957661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:25.957611    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b55cd26-1b6d-42ba-925d-84f49ab4304e" containerName="maas-api"
Nov 02 18:27:25.960943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:25.960921    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.005803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.005763    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"]
Nov 02 18:27:26.009876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.009852    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026308 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026276    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026327    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026349    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdl5\" (UniqueName: \"kubernetes.io/projected/c8df61b1-f46f-4ee7-be82-3b503d5e2969-kube-api-access-kgdl5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026453    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026648 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026483    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.026648 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026514    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026669    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026713    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026730    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.026899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026748    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62c51c1-59c7-4553-bf45-ec662a32b98e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.026899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026830    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.026899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026866    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.027047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026932    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.027047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.026960    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.027047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.027021    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9nr\" (UniqueName: \"kubernetes.io/projected/d62c51c1-59c7-4553-bf45-ec662a32b98e-kube-api-access-6q9nr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.027191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.027057    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.027191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.027157    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.027264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.027205    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.065044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.065012    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"]
Nov 02 18:27:26.116622 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.116586    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"]
Nov 02 18:27:26.128788 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.128745    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.128980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.128840    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.128980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.128939    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129051    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129108    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdl5\" (UniqueName: \"kubernetes.io/projected/c8df61b1-f46f-4ee7-be82-3b503d5e2969-kube-api-access-kgdl5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129218 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129179    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129275 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129242    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.129643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129603    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129709    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.129780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129734    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.129780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129752    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129820    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129866    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.129947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129891    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.129947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129915    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62c51c1-59c7-4553-bf45-ec662a32b98e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.129947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129943    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.130241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.129968    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.130241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130007    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.130241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130044    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.130241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130074    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9nr\" (UniqueName: \"kubernetes.io/projected/d62c51c1-59c7-4553-bf45-ec662a32b98e-kube-api-access-6q9nr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.130241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130110    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.130241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130169    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.131028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130312    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.131028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130435    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.131028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130937    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.131200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.131024    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62c51c1-59c7-4553-bf45-ec662a32b98e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.131200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.131087    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.131200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.130074    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.132169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.132135    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.132302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.132280    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.133465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.133439    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.133778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.133759    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.195989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.195905    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8df61b1-f46f-4ee7-be82-3b503d5e2969-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.198700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.198613    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62c51c1-59c7-4553-bf45-ec662a32b98e-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.199486 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.199462    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdl5\" (UniqueName: \"kubernetes.io/projected/c8df61b1-f46f-4ee7-be82-3b503d5e2969-kube-api-access-kgdl5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hlj6g\" (UID: \"c8df61b1-f46f-4ee7-be82-3b503d5e2969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.218139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.218065    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9nr\" (UniqueName: \"kubernetes.io/projected/d62c51c1-59c7-4553-bf45-ec662a32b98e-kube-api-access-6q9nr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8v2v2\" (UID: \"d62c51c1-59c7-4553-bf45-ec662a32b98e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.274818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.274779    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:26.325697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.325657    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:26.361782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.361747    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:26.559250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.559140    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c" podUID="8329047e-c2ca-4b56-bad9-a850231f5ae1" containerName="payload-pre-processing" containerID="cri-o://69b4242159e7bd334e47c5e2fc534e5eaa89632c88a535483241360ff6de3f9e" gracePeriod=25
Nov 02 18:27:26.568352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.568325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:26.721999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.721960    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"]
Nov 02 18:27:26.776227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.776197    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"]
Nov 02 18:27:26.778244 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.778189    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:26.778405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.778265    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:26.778405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.778298    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:26.787032 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:27:26.786782    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8df61b1_f46f_4ee7_be82_3b503d5e2969.slice/crio-c21dd07befc4bca516edf858682505f87bd94c65d4631a33033551f0ed536def WatchSource:0}: Error finding container c21dd07befc4bca516edf858682505f87bd94c65d4631a33033551f0ed536def: Status 404 returned error can't find the container with id c21dd07befc4bca516edf858682505f87bd94c65d4631a33033551f0ed536def
Nov 02 18:27:26.790264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.790222    4476 generic.go:356] "Generic (PLEG): container finished" podID="8329047e-c2ca-4b56-bad9-a850231f5ae1" containerID="69b4242159e7bd334e47c5e2fc534e5eaa89632c88a535483241360ff6de3f9e" exitCode=0
Nov 02 18:27:26.790466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.790344    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c" event={"ID":"8329047e-c2ca-4b56-bad9-a850231f5ae1","Type":"ContainerDied","Data":"69b4242159e7bd334e47c5e2fc534e5eaa89632c88a535483241360ff6de3f9e"}
Nov 02 18:27:26.793511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.793476    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:26.793654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.793555    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:26.793654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.793589    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:27:26.874169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.874137    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:27:26.943649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.942050    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m\" (UniqueName: \"kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m\") pod \"8329047e-c2ca-4b56-bad9-a850231f5ae1\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") "
Nov 02 18:27:26.943649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.942274    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume\") pod \"8329047e-c2ca-4b56-bad9-a850231f5ae1\" (UID: \"8329047e-c2ca-4b56-bad9-a850231f5ae1\") "
Nov 02 18:27:26.943649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.943039    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume" pod "8329047e-c2ca-4b56-bad9-a850231f5ae1" (UID: "8329047e-c2ca-4b56-bad9-a850231f5ae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:26.947407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:26.947363    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m" pod "8329047e-c2ca-4b56-bad9-a850231f5ae1" (UID: "8329047e-c2ca-4b56-bad9-a850231f5ae1"). InnerVolumeSpecName "kube-api-access-kxn7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:27.044039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.044002    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8329047e-c2ca-4b56-bad9-a850231f5ae1-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:27.044039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.044031    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxn7m\" (UniqueName: \"kubernetes.io/projected/8329047e-c2ca-4b56-bad9-a850231f5ae1-kube-api-access-kxn7m\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:27.803214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.803111    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" event={"ID":"c8df61b1-f46f-4ee7-be82-3b503d5e2969","Type":"ContainerStarted","Data":"335afd82dff6d348bf9cf95a92e685e49372368b775de1be05d542c8d1ede222"}
Nov 02 18:27:27.803214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.803167    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" event={"ID":"c8df61b1-f46f-4ee7-be82-3b503d5e2969","Type":"ContainerStarted","Data":"c21dd07befc4bca516edf858682505f87bd94c65d4631a33033551f0ed536def"}
Nov 02 18:27:27.804939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.804911    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" event={"ID":"d62c51c1-59c7-4553-bf45-ec662a32b98e","Type":"ContainerStarted","Data":"6804226de53514cb40e5183c34794ea4a24f0c8fdc6ce39cc35352a1dec929bf"}
Nov 02 18:27:27.805083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.804958    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" event={"ID":"d62c51c1-59c7-4553-bf45-ec662a32b98e","Type":"ContainerStarted","Data":"55af12655481cc69cc6c5f014d4b1a007ce3b0646f81b47fcd016e6fc0e6eae8"}
Nov 02 18:27:27.806390 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.806374    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"
Nov 02 18:27:27.806449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.806399    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c" event={"ID":"8329047e-c2ca-4b56-bad9-a850231f5ae1","Type":"ContainerDied","Data":"a682a709b402ced4bc41dcba0f6e1357b4ec5a04fa93a40313b107742690ecab"}
Nov 02 18:27:27.806491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.806447    4476 scope.go:122] "RemoveContainer" containerID="69b4242159e7bd334e47c5e2fc534e5eaa89632c88a535483241360ff6de3f9e"
Nov 02 18:27:27.902698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.892176    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:27.926049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:27.925976    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podStartSLOduration=2.925960258 podStartE2EDuration="2.925960258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:27.922275597 +0000 UTC m=+3878.762881532" watchObservedRunningTime="2026-11-02 18:27:27.925960258 +0000 UTC m=+3878.766566192"
Nov 02 18:27:28.059780 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.059686    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"]
Nov 02 18:27:28.089654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.085785    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e4b90c-7c94f4ff87-5cs4c"]
Nov 02 18:27:28.194387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.194320    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podStartSLOduration=3.19429835 podStartE2EDuration="3.19429835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:28.179342915 +0000 UTC m=+3879.019948851" watchObservedRunningTime="2026-11-02 18:27:28.19429835 +0000 UTC m=+3879.034904286"
Nov 02 18:27:28.327734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.325415    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"]
Nov 02 18:27:28.327734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.325674    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy" containerID="cri-o://cf0d9cb21bbe62daf899b048e3bf347ff0145c11dcc35091eeacd43568686d39" gracePeriod=30
Nov 02 18:27:28.327734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.326576    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:28.327734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.327372    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:28.327734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.327415    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:28.327734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.327648    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:28.329847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.329724    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:28.329847 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.329767    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:28.335762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.334702    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"]
Nov 02 18:27:28.335762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.334905    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy" containerID="cri-o://44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309" gracePeriod=30
Nov 02 18:27:28.337564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.337543    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"]
Nov 02 18:27:28.341665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.337753    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy" containerID="cri-o://80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9" gracePeriod=30
Nov 02 18:27:28.481654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:28.481020    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:29.112265 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:29.112220    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:27:29.274677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.274184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:29.278652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.276130    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:29.278652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.276183    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:29.326444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.326403    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:29.326601 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.326454    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:29.398200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.397744    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8329047e-c2ca-4b56-bad9-a850231f5ae1" path="/var/lib/kubelet/pods/8329047e-c2ca-4b56-bad9-a850231f5ae1/volumes"
Nov 02 18:27:29.625921 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.625888    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:29.950683 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:29.949108    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:30.191565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:30.181116    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:30.279773 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:30.279094    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:30.279773 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:30.279170    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:30.326285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:30.326246    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:30.326465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:30.326316    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:31.276059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:31.276016    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:31.276588 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:31.276098    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:31.328654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:31.326008    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:31.328654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:31.326089    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:32.291656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:32.276067    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:32.291656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:32.276134    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:32.326138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:32.326102    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:32.326344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:32.326164    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:33.283418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.283385    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:33.283663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.283459    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:33.326707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.326665    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:33.327112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.326734    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:33.642399 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.642371    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:27:33.704367 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.704334    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:27:33.728891 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.728864    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.728901    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.728927    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.728966    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729006    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729071    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729098    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729137    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729174    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729201    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729234    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729326    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729343    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.729372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729365    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729373    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729415    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729444    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729574    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729584    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729592    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729670    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk\") pod \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\" (UID: \"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9\") "
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729696    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729706    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.729825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.729819    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs\") pod \"7c6e0939-d38e-477a-86a8-a6218438a142\" (UID: \"7c6e0939-d38e-477a-86a8-a6218438a142\") "
Nov 02 18:27:33.730554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730317    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.730554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730338    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.730554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730354    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.730554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730371    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.730554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730386    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.730554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730401    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.730966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.730966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.730835    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.731123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.731097    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.731217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.731194    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:33.732501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.732469    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.732662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.732604    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "kube-api-access-g8t2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.733392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.733356    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:33.733670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.733622    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "kube-api-access-nlvhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.733767 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.733728    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.733907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.733887    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy" pod "7c6e0939-d38e-477a-86a8-a6218438a142" (UID: "7c6e0939-d38e-477a-86a8-a6218438a142"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:33.733907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.733891    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:33.734791 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.734765    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo" pod "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" (UID: "1c6b9573-0e7e-448b-ad22-28bdf3ca61b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:33.831845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831813    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.831845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831842    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.831845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831853    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831862    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6e0939-d38e-477a-86a8-a6218438a142-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831872    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831880    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831889    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831899    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvhk\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-kube-api-access-nlvhk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831910    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8t2c\" (UniqueName: \"kubernetes.io/projected/7c6e0939-d38e-477a-86a8-a6218438a142-kube-api-access-g8t2c\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831919    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6e0939-d38e-477a-86a8-a6218438a142-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831927    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6e0939-d38e-477a-86a8-a6218438a142-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.832086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.831935    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:33.932664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.932583    4476 generic.go:356] "Generic (PLEG): container finished" podID="7c6e0939-d38e-477a-86a8-a6218438a142" containerID="80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9" exitCode=0
Nov 02 18:27:33.933016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.932852    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" event={"ID":"7c6e0939-d38e-477a-86a8-a6218438a142","Type":"ContainerDied","Data":"80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9"}
Nov 02 18:27:33.933016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.932919    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66" event={"ID":"7c6e0939-d38e-477a-86a8-a6218438a142","Type":"ContainerDied","Data":"0ee5e6954f14e69a09bd6ce6417de143f5bfe921a08b42e02f5fa4f6066478eb"}
Nov 02 18:27:33.933016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.932953    4476 scope.go:122] "RemoveContainer" containerID="80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9"
Nov 02 18:27:33.943782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.943073    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"
Nov 02 18:27:33.954128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.953931    4476 generic.go:356] "Generic (PLEG): container finished" podID="f8056dae-a697-48e9-b811-1f60ebd61121" containerID="cf0d9cb21bbe62daf899b048e3bf347ff0145c11dcc35091eeacd43568686d39" exitCode=0
Nov 02 18:27:33.954128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.954038    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" event={"ID":"f8056dae-a697-48e9-b811-1f60ebd61121","Type":"ContainerDied","Data":"cf0d9cb21bbe62daf899b048e3bf347ff0145c11dcc35091eeacd43568686d39"}
Nov 02 18:27:33.957668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.956584    4476 generic.go:356] "Generic (PLEG): container finished" podID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerID="44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309" exitCode=0
Nov 02 18:27:33.957668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.956693    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" event={"ID":"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9","Type":"ContainerDied","Data":"44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309"}
Nov 02 18:27:33.957668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.956735    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk" event={"ID":"1c6b9573-0e7e-448b-ad22-28bdf3ca61b9","Type":"ContainerDied","Data":"e8d6631944b6af531974e292924e74e815187c60ca38c09e684b95848219a538"}
Nov 02 18:27:33.957668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.956818    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"
Nov 02 18:27:33.986388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.986353    4476 scope.go:122] "RemoveContainer" containerID="80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9"
Nov 02 18:27:33.987479 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:33.987446    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9\": container with ID starting with 80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9 not found: ID does not exist" containerID="80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9"
Nov 02 18:27:33.987604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.987495    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9"} err="failed to get container status \"80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9\": rpc error: code = NotFound desc = could not find container \"80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9\": container with ID starting with 80b3bc251945f5f7194768cc6142de9f793fe609130a48933a4c37c5f0ae49d9 not found: ID does not exist"
Nov 02 18:27:33.987604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:33.987520    4476 scope.go:122] "RemoveContainer" containerID="44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309"
Nov 02 18:27:34.005474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.002029    4476 scope.go:122] "RemoveContainer" containerID="44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309"
Nov 02 18:27:34.005474 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:34.002374    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309\": container with ID starting with 44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309 not found: ID does not exist" containerID="44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309"
Nov 02 18:27:34.005474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.002409    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309"} err="failed to get container status \"44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309\": rpc error: code = NotFound desc = could not find container \"44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309\": container with ID starting with 44e830b82cf4c918dd99847fe6e274b5befdcec7b103ef9db885036b8c6ab309 not found: ID does not exist"
Nov 02 18:27:34.017050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.017024    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:34.100240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.100205    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:27:34.127754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.127713    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"]
Nov 02 18:27:34.138731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.138693    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mfwjk"]
Nov 02 18:27:34.148166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148129    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148196    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148247    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148276    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148356    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148399    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148425    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148452    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148560    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:34.148669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148571    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket\") pod \"f8056dae-a697-48e9-b811-1f60ebd61121\" (UID: \"f8056dae-a697-48e9-b811-1f60ebd61121\") "
Nov 02 18:27:34.148669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148601    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:34.149036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.148782    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:34.149138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.149111    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:34.149138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.149121    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:34.149267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.149191    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.149267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.149216    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8056dae-a697-48e9-b811-1f60ebd61121-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.149267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.149230    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.151665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.151607    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "kube-api-access-6bns5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:34.151796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.151720    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:34.151796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.151736    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:34.151796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.151749    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo" pod "f8056dae-a697-48e9-b811-1f60ebd61121" (UID: "f8056dae-a697-48e9-b811-1f60ebd61121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:34.225444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.225400    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:34.245073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.245040    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"]
Nov 02 18:27:34.248797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.248769    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6kb66"]
Nov 02 18:27:34.250098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.250065    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.250196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.250104    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.250196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.250123    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8056dae-a697-48e9-b811-1f60ebd61121-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.250196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.250136    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.250196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.250150    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bns5\" (UniqueName: \"kubernetes.io/projected/f8056dae-a697-48e9-b811-1f60ebd61121-kube-api-access-6bns5\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.250196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.250165    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8056dae-a697-48e9-b811-1f60ebd61121-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:34.276135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.276097    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 02 18:27:34.276259 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.276163    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 02 18:27:34.326580 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.326544    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 02 18:27:34.326792 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.326613    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 02 18:27:34.970112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.970079    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5" event={"ID":"f8056dae-a697-48e9-b811-1f60ebd61121","Type":"ContainerDied","Data":"0f5054e4aa35faa0273503c85c27c911c39c1892de70804d2cdbc68485e18284"}
Nov 02 18:27:34.970340 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.970094    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"
Nov 02 18:27:34.970340 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:34.970142    4476 scope.go:122] "RemoveContainer" containerID="cf0d9cb21bbe62daf899b048e3bf347ff0145c11dcc35091eeacd43568686d39"
Nov 02 18:27:35.029391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:35.029355    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:35.139070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:35.139032    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"]
Nov 02 18:27:35.186790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:35.186751    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-6rkp5"]
Nov 02 18:27:35.396327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:35.396284    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" path="/var/lib/kubelet/pods/1c6b9573-0e7e-448b-ad22-28bdf3ca61b9/volumes"
Nov 02 18:27:35.397005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:35.396983    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" path="/var/lib/kubelet/pods/7c6e0939-d38e-477a-86a8-a6218438a142/volumes"
Nov 02 18:27:35.397648 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:35.397610    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" path="/var/lib/kubelet/pods/f8056dae-a697-48e9-b811-1f60ebd61121/volumes"
Nov 02 18:27:36.251273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.251241    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:36.251713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.251661    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:36.253115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.253082    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g"
Nov 02 18:27:36.277121 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.276925    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:27:36.278652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.278114    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:27:36.307642 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.307108    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:36.308365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.308296    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:36.309140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:36.309122    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2"
Nov 02 18:27:37.266014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:37.265978    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:37.487382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:37.487333    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:39.834835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:39.834799    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:41.415824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.415791    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:41.599754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.599723    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"]
Nov 02 18:27:41.640530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.640503    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:41.650442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.650417    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a9496c-4bh82"]
Nov 02 18:27:41.837792 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.837702    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"]
Nov 02 18:27:41.838311 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.838282    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy" containerID="cri-o://40699ed76f6ffba389edcb510cf8f9a35f33b417019ed0f7c2df5cffc21bce4b" gracePeriod=30
Nov 02 18:27:41.925302 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:41.925270    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:42.712888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:42.712843    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:43.385662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:43.385599    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2698860-cdd0-42dd-bf71-356cc6072414" path="/var/lib/kubelet/pods/b2698860-cdd0-42dd-bf71-356cc6072414/volumes"
Nov 02 18:27:46.080537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:46.080496    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:47.079225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.079191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:47.110483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.109506    4476 generic.go:356] "Generic (PLEG): container finished" podID="144aff60-b302-4731-9031-da37c7aebea0" containerID="40699ed76f6ffba389edcb510cf8f9a35f33b417019ed0f7c2df5cffc21bce4b" exitCode=0
Nov 02 18:27:47.110483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.109590    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" event={"ID":"144aff60-b302-4731-9031-da37c7aebea0","Type":"ContainerDied","Data":"40699ed76f6ffba389edcb510cf8f9a35f33b417019ed0f7c2df5cffc21bce4b"}
Nov 02 18:27:47.110483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.109717    4476 scope.go:122] "RemoveContainer" containerID="445b2a58d2a5908721647d1d44f1268cebdc691456bb043d4a72a436185d8c58"
Nov 02 18:27:47.188484 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.188457    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:27:47.277618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.277580    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:47.333360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333325    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333421    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333513    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333541    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333575    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333611    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333749    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333777    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333840    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt\") pod \"144aff60-b302-4731-9031-da37c7aebea0\" (UID: \"144aff60-b302-4731-9031-da37c7aebea0\") "
Nov 02 18:27:47.333933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.333851    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:47.334263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.334191    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:47.334369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.334315    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.334369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.334336    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.335235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.335176    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:47.335461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.335427    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:47.335755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.335714    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:47.336390 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.336365    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:47.337603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.337578    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:47.337988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.337967    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "kube-api-access-7hwdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:47.338794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.338775    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo" pod "144aff60-b302-4731-9031-da37c7aebea0" (UID: "144aff60-b302-4731-9031-da37c7aebea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:47.435908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435866    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.435908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435900    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/144aff60-b302-4731-9031-da37c7aebea0-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.435908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435909    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.436190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435920    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/144aff60-b302-4731-9031-da37c7aebea0-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.436190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435929    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.436190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435938    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwdt\" (UniqueName: \"kubernetes.io/projected/144aff60-b302-4731-9031-da37c7aebea0-kube-api-access-7hwdt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.436190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:47.435959    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/144aff60-b302-4731-9031-da37c7aebea0-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:48.042996 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.042957    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:48.118031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.117996    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"
Nov 02 18:27:48.118484 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.118037    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq" event={"ID":"144aff60-b302-4731-9031-da37c7aebea0","Type":"ContainerDied","Data":"1cb5efe323815127ea1b46dee387c1e938a0e81392a76924c60f5089a687bc06"}
Nov 02 18:27:48.118484 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.118103    4476 scope.go:122] "RemoveContainer" containerID="40699ed76f6ffba389edcb510cf8f9a35f33b417019ed0f7c2df5cffc21bce4b"
Nov 02 18:27:48.239223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.239190    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"]
Nov 02 18:27:48.242932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.242903    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cd738b-pvkjk"]
Nov 02 18:27:48.275485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.275445    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:48.348550 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.348456    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"]
Nov 02 18:27:48.352712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.352673    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a9496c-openshift-default-6875d76469-f5rbq"]
Nov 02 18:27:48.450701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.450663    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"]
Nov 02 18:27:48.450965 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.450937    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy" containerID="cri-o://edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087" gracePeriod=30
Nov 02 18:27:48.547370 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:48.547334    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:49.257267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:49.257175    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:49.393264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:49.393226    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144aff60-b302-4731-9031-da37c7aebea0" path="/var/lib/kubelet/pods/144aff60-b302-4731-9031-da37c7aebea0/volumes"
Nov 02 18:27:49.393734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:49.393718    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e" path="/var/lib/kubelet/pods/fe3a05a4-94a9-44b9-8b82-d7c8cc80ef3e/volumes"
Nov 02 18:27:49.500332 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:49.500297    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"]
Nov 02 18:27:49.594989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:49.594904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:49.807251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:49.807190    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:50.006479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:50.006421    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:50.056768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:50.056743    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:27:50.065305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:50.065275    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:27:51.160593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:51.160542    4476 scope.go:122] "RemoveContainer" containerID="ce33e512c8a8ef4bb2f33e8876f68e0b37be3e4407c5d4915587eebca7a801ea"
Nov 02 18:27:53.420061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:53.420021    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:53.747960 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:53.747855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:54.144592 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.144559    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:27:54.174912 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.174872    4476 generic.go:356] "Generic (PLEG): container finished" podID="263b056f-8821-4407-b9cc-b4201d0a9952" containerID="edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087" exitCode=0
Nov 02 18:27:54.175124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.174981    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"
Nov 02 18:27:54.175124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.175007    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" event={"ID":"263b056f-8821-4407-b9cc-b4201d0a9952","Type":"ContainerDied","Data":"edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087"}
Nov 02 18:27:54.175124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.175066    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n" event={"ID":"263b056f-8821-4407-b9cc-b4201d0a9952","Type":"ContainerDied","Data":"e1a22c8cfe5250afc8759365dff31ae7cffcf5f32b7dc485018ec0418069024c"}
Nov 02 18:27:54.175124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.175098    4476 scope.go:122] "RemoveContainer" containerID="edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087"
Nov 02 18:27:54.186038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.186013    4476 scope.go:122] "RemoveContainer" containerID="2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"
Nov 02 18:27:54.196414 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.196394    4476 scope.go:122] "RemoveContainer" containerID="edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087"
Nov 02 18:27:54.196767 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:54.196746    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087\": container with ID starting with edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087 not found: ID does not exist" containerID="edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087"
Nov 02 18:27:54.196878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.196775    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087"} err="failed to get container status \"edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087\": rpc error: code = NotFound desc = could not find container \"edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087\": container with ID starting with edf2cc89cfc9868aba8e943b933132220a790b5fc832448038ac8522bc261087 not found: ID does not exist"
Nov 02 18:27:54.196878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.196795    4476 scope.go:122] "RemoveContainer" containerID="2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"
Nov 02 18:27:54.197072 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:54.197054    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6\": container with ID starting with 2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6 not found: ID does not exist" containerID="2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"
Nov 02 18:27:54.197120 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.197075    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6"} err="failed to get container status \"2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6\": rpc error: code = NotFound desc = could not find container \"2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6\": container with ID starting with 2b0580af3dce873a94a807337f870003525edcb3810963af3c64273a8d3e3bb6 not found: ID does not exist"
Nov 02 18:27:54.215477 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215449    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.215643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215506    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.215643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215537    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.215742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215659    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.215742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215721    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.215869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215786    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.215869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215830    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.215869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215842    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.215869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215862    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.216113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215903    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.216113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215931    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.216113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.215988    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs\") pod \"263b056f-8821-4407-b9cc-b4201d0a9952\" (UID: \"263b056f-8821-4407-b9cc-b4201d0a9952\") "
Nov 02 18:27:54.216356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216289    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:54.216436 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216319    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.216816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216787    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b056f-8821-4407-b9cc-b4201d0a9952-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.216950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216818    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.216950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216834    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.216950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216848    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.216950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.216863    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.217939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.217917    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:54.218217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.218190    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "kube-api-access-r46n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:54.218297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.218223    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.218297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.218223    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token" pod "263b056f-8821-4407-b9cc-b4201d0a9952" (UID: "263b056f-8821-4407-b9cc-b4201d0a9952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:54.317937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.317846    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b056f-8821-4407-b9cc-b4201d0a9952-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.317937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.317886    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-kube-api-access-r46n7\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.317937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.317903    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b056f-8821-4407-b9cc-b4201d0a9952-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.317937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.317918    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b056f-8821-4407-b9cc-b4201d0a9952-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.501106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.501064    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7" podUID="5bb24fbe-d20c-4690-ae54-b564288a80b7" containerName="payload-pre-processing" containerID="cri-o://7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f" gracePeriod=25
Nov 02 18:27:54.597086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.596998    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"]
Nov 02 18:27:54.601421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.601393    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cd738b-openshift-default-787d5f4669-gsj4n"]
Nov 02 18:27:54.783647 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.783595    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:27:54.924422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.924382    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl\" (UniqueName: \"kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl\") pod \"5bb24fbe-d20c-4690-ae54-b564288a80b7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") "
Nov 02 18:27:54.924673 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.924528    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume\") pod \"5bb24fbe-d20c-4690-ae54-b564288a80b7\" (UID: \"5bb24fbe-d20c-4690-ae54-b564288a80b7\") "
Nov 02 18:27:54.925017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.924990    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume" pod "5bb24fbe-d20c-4690-ae54-b564288a80b7" (UID: "5bb24fbe-d20c-4690-ae54-b564288a80b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:54.927095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:54.927056    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl" pod "5bb24fbe-d20c-4690-ae54-b564288a80b7" (UID: "5bb24fbe-d20c-4690-ae54-b564288a80b7"). InnerVolumeSpecName "kube-api-access-srjzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:55.026400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.026367    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjzl\" (UniqueName: \"kubernetes.io/projected/5bb24fbe-d20c-4690-ae54-b564288a80b7-kube-api-access-srjzl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:55.026400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.026399    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb24fbe-d20c-4690-ae54-b564288a80b7-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:27:55.168255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.168222    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:55.185167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.185071    4476 generic.go:356] "Generic (PLEG): container finished" podID="5bb24fbe-d20c-4690-ae54-b564288a80b7" containerID="7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f" exitCode=0
Nov 02 18:27:55.185167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.185122    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"
Nov 02 18:27:55.185167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.185150    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7" event={"ID":"5bb24fbe-d20c-4690-ae54-b564288a80b7","Type":"ContainerDied","Data":"7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f"}
Nov 02 18:27:55.185441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.185210    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7" event={"ID":"5bb24fbe-d20c-4690-ae54-b564288a80b7","Type":"ContainerDied","Data":"1323cf89b421258cb1cfdb770e6316bc9d5aa87d8355c497d3c03704c7ea7f62"}
Nov 02 18:27:55.185441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.185238    4476 scope.go:122] "RemoveContainer" containerID="7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f"
Nov 02 18:27:55.196564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.196541    4476 scope.go:122] "RemoveContainer" containerID="7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f"
Nov 02 18:27:55.196980 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:55.196963    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f\": container with ID starting with 7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f not found: ID does not exist" containerID="7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f"
Nov 02 18:27:55.197037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.196989    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f"} err="failed to get container status \"7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f\": rpc error: code = NotFound desc = could not find container \"7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f\": container with ID starting with 7a7bdb57f0fd66986a1129c66bceeea5895e29a625ecd835ee5d04996ba4d02f not found: ID does not exist"
Nov 02 18:27:55.351836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.351785    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"]
Nov 02 18:27:55.355380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.355353    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a72f55-7d46fdccff-cnfp7"]
Nov 02 18:27:55.385731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.385691    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" path="/var/lib/kubelet/pods/263b056f-8821-4407-b9cc-b4201d0a9952/volumes"
Nov 02 18:27:55.386213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.386191    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb24fbe-d20c-4690-ae54-b564288a80b7" path="/var/lib/kubelet/pods/5bb24fbe-d20c-4690-ae54-b564288a80b7/volumes"
Nov 02 18:27:55.712572 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:55.712540    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:56.964476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:56.964437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:57.183720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:57.183684    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:58.576322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:27:58.576284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:59.061084 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:27:59.061049    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:28:01.430263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:01.430191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:02.033881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:02.033851    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:02.826924 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:02.826886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:04.741181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:04.741140    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:07.158310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:07.158275    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:07.352304 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:07.352272    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:09.250810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:09.250765    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:09.729802 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:09.729768    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:09.922744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:09.922716    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:10.670251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:10.670215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:11.138313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.138257    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"]
Nov 02 18:28:11.149451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.149420    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy"
Nov 02 18:28:11.149708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.149691    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy"
Nov 02 18:28:11.149832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.149821    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy"
Nov 02 18:28:11.149917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.149904    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy"
Nov 02 18:28:11.150021 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150010    4476 cpu_manager.go:414] "containerMap: removing container" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy"
Nov 02 18:28:11.150098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150088    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy"
Nov 02 18:28:11.150185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150176    4476 cpu_manager.go:414] "containerMap: removing container" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy"
Nov 02 18:28:11.150260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150251    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy"
Nov 02 18:28:11.150365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150355    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5bb24fbe-d20c-4690-ae54-b564288a80b7" containerName="payload-pre-processing"
Nov 02 18:28:11.150442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150432    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb24fbe-d20c-4690-ae54-b564288a80b7" containerName="payload-pre-processing"
Nov 02 18:28:11.150532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150522    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8329047e-c2ca-4b56-bad9-a850231f5ae1" containerName="payload-pre-processing"
Nov 02 18:28:11.150607 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150598    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8329047e-c2ca-4b56-bad9-a850231f5ae1" containerName="payload-pre-processing"
Nov 02 18:28:11.150714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150702    4476 cpu_manager.go:414] "containerMap: removing container" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy"
Nov 02 18:28:11.150798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150787    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy"
Nov 02 18:28:11.150893 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150883    4476 cpu_manager.go:414] "containerMap: removing container" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy"
Nov 02 18:28:11.150981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.150970    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy"
Nov 02 18:28:11.151064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151054    4476 cpu_manager.go:414] "containerMap: removing container" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy"
Nov 02 18:28:11.151142 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151132    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy"
Nov 02 18:28:11.151396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151382    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb24fbe-d20c-4690-ae54-b564288a80b7" containerName="payload-pre-processing"
Nov 02 18:28:11.151502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151491    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8329047e-c2ca-4b56-bad9-a850231f5ae1" containerName="payload-pre-processing"
Nov 02 18:28:11.151684 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151670    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6b9573-0e7e-448b-ad22-28bdf3ca61b9" containerName="istio-proxy"
Nov 02 18:28:11.151785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151775    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy"
Nov 02 18:28:11.151871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151862    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy"
Nov 02 18:28:11.151965 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.151955    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6e0939-d38e-477a-86a8-a6218438a142" containerName="istio-proxy"
Nov 02 18:28:11.152049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.152040    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8056dae-a697-48e9-b811-1f60ebd61121" containerName="istio-proxy"
Nov 02 18:28:11.160014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.159989    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204037    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204097    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204140    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204176    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204220    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204248    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdlc\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204450    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.207671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.204484    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.208772 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.208254    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.225879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.225847    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"]
Nov 02 18:28:11.267711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.267676    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-740c43-openshift-default-dockercfg-nhf27\""
Nov 02 18:28:11.292446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.292409    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:11.310136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310097    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310163    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310210    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310251    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310299    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310329    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdlc\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310420    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310457    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.310568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.310497    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.311876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.311849    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.314288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.312344    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.314456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.312577    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.314563 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.313224    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.314884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.314850    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.316242 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.315866    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.316934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.316911    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.368420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.368387    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.370929 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.370899    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdlc\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc\") pod \"e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") " pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.510581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.510492    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:11.618442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.615324    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:11.960126 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.960102    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"]
Nov 02 18:28:11.962535 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:28:11.962511    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bb92d5_dd8a_4c83_bd9b_e8d1f54af9a9.slice/crio-2e1182459f9a13032f7aee4450471fbf73fcba292fd8dc0a11751e4309688bf2 WatchSource:0}: Error finding container 2e1182459f9a13032f7aee4450471fbf73fcba292fd8dc0a11751e4309688bf2: Status 404 returned error can't find the container with id 2e1182459f9a13032f7aee4450471fbf73fcba292fd8dc0a11751e4309688bf2
Nov 02 18:28:11.965719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.965685    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:28:11.965827 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.965768    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:28:11.965827 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:11.965813    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:28:12.316392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:12.316298    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" event={"ID":"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9","Type":"ContainerStarted","Data":"273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"}
Nov 02 18:28:12.316392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:12.316363    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" event={"ID":"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9","Type":"ContainerStarted","Data":"2e1182459f9a13032f7aee4450471fbf73fcba292fd8dc0a11751e4309688bf2"}
Nov 02 18:28:12.436291 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:12.436240    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podStartSLOduration=1.4362230409999999 podStartE2EDuration="1.436223041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:12.434059398 +0000 UTC m=+3923.274665333" watchObservedRunningTime="2026-11-02 18:28:12.436223041 +0000 UTC m=+3923.276828976"
Nov 02 18:28:12.511528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:12.511488    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:13.517806 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:13.517773    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:14.329797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:14.329759    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:14.330790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:14.330768    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:28:14.765128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:14.765088    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:15.247515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:15.247481    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:15.457355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:15.457325    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:18.504388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:18.504359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:18.698323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:18.698277    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:20.158595 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:20.158553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:25.297650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:25.297385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:29.061251 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:28:29.061205    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:28:30.057079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:30.057042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:34.779989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:34.779831    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:34.969105 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:34.969066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:35.173940 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:35.173907    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:39.864583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:39.864531    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:44.680535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:44.680499    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:49.374286 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:49.374226    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:54.170219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:54.170163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:58.828939 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:58.828896    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:59.012735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:28:59.012699    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:59.061502 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:28:59.061460    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:29:02.375862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:02.375820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:03.779141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:03.779109    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:08.561127 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:08.561089    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:12.679768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:12.679729    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:29:12.680213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:12.679790    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:29:12.798019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:12.797986    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:14.597395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:14.597354    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:14.783822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:14.783778    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:15.065678 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.065535    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:15.846375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.846341    4476 generic.go:356] "Generic (PLEG): container finished" podID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerID="426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a" exitCode=0
Nov 02 18:29:15.846875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.846413    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" event={"ID":"7d55225d-9bad-48ad-a7d8-3f0a741ec985","Type":"ContainerDied","Data":"426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"}
Nov 02 18:29:15.846875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.846845    4476 scope.go:122] "RemoveContainer" containerID="426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"
Nov 02 18:29:15.848435 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.848403    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:29:15.848523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.848497    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:29:15.848569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.848544    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:29:15.956732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:15.956694    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:16.857651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:16.855306    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:16.896205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:16.896168    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" event={"ID":"7d55225d-9bad-48ad-a7d8-3f0a741ec985","Type":"ContainerStarted","Data":"22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3"}
Nov 02 18:29:17.100115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:17.100081    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:17.688777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:17.688743    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:29:17.689000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:17.688979    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 18:29:17.689061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:17.689037    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 18:29:18.688323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:18.688289    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 18:29:18.688782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:18.688350    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 18:29:19.689119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:19.689081    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 18:29:19.689695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:19.689169    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 18:29:19.952762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:19.952662    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:20.688903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:20.688851    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 18:29:20.689067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:20.688925    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 18:29:21.333139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:21.333093    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:21.570084 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:21.570051    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:21.688985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:21.688946    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Nov 02 18:29:21.689190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:21.689008    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Nov 02 18:29:22.692687 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:22.692656    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:29:22.693087 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:22.692937    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:29:22.693677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:22.693660    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:29:22.746677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:22.746647    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:22.957235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:22.957156    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:24.553271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:24.553236    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:24.745817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:24.745781    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:25.410235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:25.410190    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:29.065365 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:29:29.065318    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:29:30.207552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:30.207506    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:34.647094 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.647051    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"]
Nov 02 18:29:34.648133 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.648111    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="263b056f-8821-4407-b9cc-b4201d0a9952" containerName="istio-proxy"
Nov 02 18:29:34.648257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.648139    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="144aff60-b302-4731-9031-da37c7aebea0" containerName="istio-proxy"
Nov 02 18:29:34.651114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.651093    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.706886 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.706856    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"]
Nov 02 18:29:34.751721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.751691    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-740c43-serving-cert\""
Nov 02 18:29:34.786267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.786229    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.786575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.786555    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.786736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.786719    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxkn\" (UniqueName: \"kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.786980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.786964    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.883411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.883380    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:34.889041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.889005    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.889265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.889103    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.889265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.889174    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.889265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.889205    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxkn\" (UniqueName: \"kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.889265 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:29:34.889244    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-serving-cert: secret "maas-api-e2e-inf-a-740c43-serving-cert" not found
Nov 02 18:29:34.889494 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:29:34.889320    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls podName:585ed752-b048-4c68-9713-2892dad85ad5 nodeName:}" failed. No retries permitted until 2026-11-02 18:29:35.389303406 +0000 UTC m=+4006.229909319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls") pod "maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" (UID: "585ed752-b048-4c68-9713-2892dad85ad5") : secret "maas-api-e2e-inf-a-740c43-serving-cert" not found
Nov 02 18:29:34.889913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.889888    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.892516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.892493    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:34.946167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:34.946091    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxkn\" (UniqueName: \"kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:35.073962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.073929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:35.257257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.257174    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:35.394134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.394105    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:35.396674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.396649    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\") pod \"maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:35.565198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.565112    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:35.950466 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:29:35.950432    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585ed752_b048_4c68_9713_2892dad85ad5.slice/crio-16ccda3f6b24f36044b9e2684db9020989d1065214d322493678e6f135146b82 WatchSource:0}: Error finding container 16ccda3f6b24f36044b9e2684db9020989d1065214d322493678e6f135146b82: Status 404 returned error can't find the container with id 16ccda3f6b24f36044b9e2684db9020989d1065214d322493678e6f135146b82
Nov 02 18:29:35.950938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.950593    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"]
Nov 02 18:29:35.952444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:35.952420    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:29:36.034574 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:36.034538    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" event={"ID":"585ed752-b048-4c68-9713-2892dad85ad5","Type":"ContainerStarted","Data":"16ccda3f6b24f36044b9e2684db9020989d1065214d322493678e6f135146b82"}
Nov 02 18:29:37.045104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:37.045004    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" event={"ID":"585ed752-b048-4c68-9713-2892dad85ad5","Type":"ContainerStarted","Data":"84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf"}
Nov 02 18:29:37.045577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:37.045113    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:37.163119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:37.163061    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" podStartSLOduration=2.362467336 podStartE2EDuration="3.16304588s" totalImagesPullingTime="800.578544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:37.159693564 +0000 UTC m=+4008.000299500" watchObservedRunningTime="2026-11-02 18:29:37.16304588 +0000 UTC m=+4008.003651814"
Nov 02 18:29:39.923359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:39.923320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:43.061156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:43.061123    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:29:44.780217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:44.780184    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:49.462429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:49.462385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:54.189129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:54.189097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:58.998651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:29:58.998592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:59.068546 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:29:59.068505    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:30:00.172578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.168728    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"]
Nov 02 18:30:00.176033 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.176004    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.178355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.178323    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"]
Nov 02 18:30:00.182986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.182968    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.196859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.196837    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"]
Nov 02 18:30:00.200462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.200445    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"]
Nov 02 18:30:00.200586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.200569    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.203844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.203829    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.231272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.231247    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"]
Nov 02 18:30:00.233722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.233705    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:30:00.235479 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.235461    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"]
Nov 02 18:30:00.235558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.235547    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.247521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.247497    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"]
Nov 02 18:30:00.251169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.251150    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.285263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.285230    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"]
Nov 02 18:30:00.286282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.286248    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"]
Nov 02 18:30:00.303360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303317    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9j4\" (UniqueName: \"kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.303578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303387    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.303578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303459    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfvw\" (UniqueName: \"kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw\") pod \"maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.303578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303488    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.303578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303550    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87w8z\" (UniqueName: \"kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.303916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303600    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.303916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303645    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrpf\" (UniqueName: \"kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.303916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303682    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcb8g\" (UniqueName: \"kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.303916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303714    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.303916 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.303814    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.357693 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.337708    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"]
Nov 02 18:30:00.368371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.368332    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"]
Nov 02 18:30:00.379265 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.379229    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"]
Nov 02 18:30:00.404731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404697    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.404901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404766    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smrzp\" (UniqueName: \"kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp\") pod \"maas-api-key-cleanup-29894070-scm9f\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.404901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404797    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-scm9f\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.404901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404821    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9j4\" (UniqueName: \"kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.405025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404912    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.405025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404965    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfvw\" (UniqueName: \"kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw\") pod \"maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.405025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.404993    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.405164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405059    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87w8z\" (UniqueName: \"kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.405164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405106    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.405164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405131    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrpf\" (UniqueName: \"kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.405328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405163    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcb8g\" (UniqueName: \"kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.405328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405194    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.405657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405615    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.405789 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405765    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.405895 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405879    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.405965 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405921    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.406007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.405978    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.466160 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.466079    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87w8z\" (UniqueName: \"kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.466739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.466701    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcb8g\" (UniqueName: \"kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.466878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.466803    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfvw\" (UniqueName: \"kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw\") pod \"maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.467135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.467112    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrpf\" (UniqueName: \"kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.467202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.467147    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9j4\" (UniqueName: \"kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.490974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.490949    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:00.500948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.500923    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:00.506208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.506186    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrzp\" (UniqueName: \"kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp\") pod \"maas-api-key-cleanup-29894070-scm9f\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.506318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.506223    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-scm9f\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.507047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.507024    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-scm9f\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.516279 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.516259    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:00.526217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.523283    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:00.563214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.552828    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:00.574386 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.574346    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrzp\" (UniqueName: \"kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp\") pod \"maas-api-key-cleanup-29894070-scm9f\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.588054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.587605    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:00.925254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.925212    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"]
Nov 02 18:30:00.925706 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:30:00.925681    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74db920_62da_484a_8190_1eb3f793166e.slice/crio-676ae0666aba02744b7ab800ce5ce2d4df5506c1f627bef140c3e62863dcd1dd WatchSource:0}: Error finding container 676ae0666aba02744b7ab800ce5ce2d4df5506c1f627bef140c3e62863dcd1dd: Status 404 returned error can't find the container with id 676ae0666aba02744b7ab800ce5ce2d4df5506c1f627bef140c3e62863dcd1dd
Nov 02 18:30:00.985269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:00.985243    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"]
Nov 02 18:30:01.032491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.032463    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"]
Nov 02 18:30:01.033981 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:30:01.033956    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf007f79b_bbf9_4571_af63_eee3b962e409.slice/crio-0516d411a68d4454d234a7a620f6bad56aa7bebbdbe7f46298fc0a273a282865 WatchSource:0}: Error finding container 0516d411a68d4454d234a7a620f6bad56aa7bebbdbe7f46298fc0a273a282865: Status 404 returned error can't find the container with id 0516d411a68d4454d234a7a620f6bad56aa7bebbdbe7f46298fc0a273a282865
Nov 02 18:30:01.065148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.056191    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"]
Nov 02 18:30:01.099233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.099207    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"]
Nov 02 18:30:01.102514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.102493    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"]
Nov 02 18:30:01.226586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.226549    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerStarted","Data":"992a04932d7fd4dd78e409bd47d0bad4400713dcaa65d4cae6ee5e7adcd2e443"}
Nov 02 18:30:01.227843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.227819    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerStarted","Data":"5989c5c82c1e600523e2a35e0411fadfbb474f0e32be527ba0b681a128034417"}
Nov 02 18:30:01.229112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.229084    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerStarted","Data":"0516d411a68d4454d234a7a620f6bad56aa7bebbdbe7f46298fc0a273a282865"}
Nov 02 18:30:01.230313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.230289    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerStarted","Data":"69293040af866bed3f2ba1ca28c7e7c0d26c70623c5ebce69aa71a7a1e40fc4a"}
Nov 02 18:30:01.231386 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.231367    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" event={"ID":"2f8196cd-5d00-4e27-8c1c-a217bc800de9","Type":"ContainerStarted","Data":"0b56dcaff5c1a5fa86158d4eba864bceba6be1b9cbcb562b548e851a0a0dd4c1"}
Nov 02 18:30:01.232460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:01.232443    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" event={"ID":"f74db920-62da-484a-8190-1eb3f793166e","Type":"ContainerStarted","Data":"676ae0666aba02744b7ab800ce5ce2d4df5506c1f627bef140c3e62863dcd1dd"}
Nov 02 18:30:02.271556 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.269043    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerStarted","Data":"1113c1f837cbd6c8b54a922921bc48115ff45462b0d5aab0f1b386aba270a48e"}
Nov 02 18:30:02.272841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.272811    4476 generic.go:356] "Generic (PLEG): container finished" podID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerID="1cbfef6b47b789fd50bf5ad76c581595abf4aa3ea3e0a7b38cb5b0520b4d4251" exitCode=60
Nov 02 18:30:02.273000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.272915    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" event={"ID":"2f8196cd-5d00-4e27-8c1c-a217bc800de9","Type":"ContainerDied","Data":"1cbfef6b47b789fd50bf5ad76c581595abf4aa3ea3e0a7b38cb5b0520b4d4251"}
Nov 02 18:30:02.273206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.273184    4476 scope.go:122] "RemoveContainer" containerID="1cbfef6b47b789fd50bf5ad76c581595abf4aa3ea3e0a7b38cb5b0520b4d4251"
Nov 02 18:30:02.275180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.275047    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" event={"ID":"f74db920-62da-484a-8190-1eb3f793166e","Type":"ContainerDied","Data":"9c33c0d8894b71ddb147ca11847f3599a5110e68c6f729a467b7804d44321700"}
Nov 02 18:30:02.275333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.275190    4476 scope.go:122] "RemoveContainer" containerID="9c33c0d8894b71ddb147ca11847f3599a5110e68c6f729a467b7804d44321700"
Nov 02 18:30:02.275451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.274939    4476 generic.go:356] "Generic (PLEG): container finished" podID="f74db920-62da-484a-8190-1eb3f793166e" containerID="9c33c0d8894b71ddb147ca11847f3599a5110e68c6f729a467b7804d44321700" exitCode=60
Nov 02 18:30:02.279247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.279023    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerStarted","Data":"3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9"}
Nov 02 18:30:02.279247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.279111    4476 scope.go:122] "RemoveContainer" containerID="3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9"
Nov 02 18:30:02.280843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.280820    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerStarted","Data":"1eb4333a4668f5d3dc02b6c236cc3f5b1bdc6c71f5ea65d88948e2ef2265bcd7"}
Nov 02 18:30:02.281143 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.281110    4476 scope.go:122] "RemoveContainer" containerID="1eb4333a4668f5d3dc02b6c236cc3f5b1bdc6c71f5ea65d88948e2ef2265bcd7"
Nov 02 18:30:02.282735 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.282713    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerStarted","Data":"71ce6e82127a18b7dcdbeaee40c0a46ddbec8a11fb52d7454ca68c46d6463d70"}
Nov 02 18:30:02.383165 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:02.383132    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a6f3a3_a72b_4705_9087_4211859f82a0.slice/crio-conmon-3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a6f3a3_a72b_4705_9087_4211859f82a0.slice/crio-3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d9eb84_501f_4f3a_b68a_ae84f6aa86b4.slice/crio-1eb4333a4668f5d3dc02b6c236cc3f5b1bdc6c71f5ea65d88948e2ef2265bcd7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd833b6_aeb3_45ca_a6a4_cf2bdab12827.slice/crio-1113c1f837cbd6c8b54a922921bc48115ff45462b0d5aab0f1b386aba270a48e.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:30:02.388372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.388323    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" podStartSLOduration=2.388308793 podStartE2EDuration="2.388308793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:02.382651272 +0000 UTC m=+4033.223257208" watchObservedRunningTime="2026-11-02 18:30:02.388308793 +0000 UTC m=+4033.228914787"
Nov 02 18:30:02.728401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.728303    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:02.869687 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:02.869613    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" podStartSLOduration=2.8695975970000003 podStartE2EDuration="2.869597597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:02.85396227 +0000 UTC m=+4033.694568204" watchObservedRunningTime="2026-11-02 18:30:02.869597597 +0000 UTC m=+4033.710203522"
Nov 02 18:30:03.293066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.293030    4476 generic.go:356] "Generic (PLEG): container finished" podID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerID="3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9" exitCode=60
Nov 02 18:30:03.293502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.293160    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerDied","Data":"3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9"}
Nov 02 18:30:03.293502 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.293198    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerStarted","Data":"c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79"}
Nov 02 18:30:03.297387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.296534    4476 generic.go:356] "Generic (PLEG): container finished" podID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerID="1eb4333a4668f5d3dc02b6c236cc3f5b1bdc6c71f5ea65d88948e2ef2265bcd7" exitCode=60
Nov 02 18:30:03.297387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.296612    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerDied","Data":"1eb4333a4668f5d3dc02b6c236cc3f5b1bdc6c71f5ea65d88948e2ef2265bcd7"}
Nov 02 18:30:03.297387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.296711    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerStarted","Data":"a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2"}
Nov 02 18:30:03.298841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.298817    4476 generic.go:356] "Generic (PLEG): container finished" podID="f007f79b-bbf9-4571-af63-eee3b962e409" containerID="71ce6e82127a18b7dcdbeaee40c0a46ddbec8a11fb52d7454ca68c46d6463d70" exitCode=60
Nov 02 18:30:03.298936 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.298916    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerDied","Data":"71ce6e82127a18b7dcdbeaee40c0a46ddbec8a11fb52d7454ca68c46d6463d70"}
Nov 02 18:30:03.299242 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.299222    4476 scope.go:122] "RemoveContainer" containerID="71ce6e82127a18b7dcdbeaee40c0a46ddbec8a11fb52d7454ca68c46d6463d70"
Nov 02 18:30:03.301554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.300599    4476 generic.go:356] "Generic (PLEG): container finished" podID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerID="1113c1f837cbd6c8b54a922921bc48115ff45462b0d5aab0f1b386aba270a48e" exitCode=60
Nov 02 18:30:03.301554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.300754    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerDied","Data":"1113c1f837cbd6c8b54a922921bc48115ff45462b0d5aab0f1b386aba270a48e"}
Nov 02 18:30:03.301554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.301091    4476 scope.go:122] "RemoveContainer" containerID="1113c1f837cbd6c8b54a922921bc48115ff45462b0d5aab0f1b386aba270a48e"
Nov 02 18:30:03.305554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.305534    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" event={"ID":"2f8196cd-5d00-4e27-8c1c-a217bc800de9","Type":"ContainerStarted","Data":"a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea"}
Nov 02 18:30:03.308153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.308130    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" event={"ID":"f74db920-62da-484a-8190-1eb3f793166e","Type":"ContainerStarted","Data":"aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70"}
Nov 02 18:30:03.411832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.411760    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" podStartSLOduration=3.411699047 podStartE2EDuration="3.411699047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.410966106 +0000 UTC m=+4034.251572039" watchObservedRunningTime="2026-11-02 18:30:03.411699047 +0000 UTC m=+4034.252304982"
Nov 02 18:30:03.530454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.530393    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" podStartSLOduration=3.530378411 podStartE2EDuration="3.530378411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.529125937 +0000 UTC m=+4034.369731878" watchObservedRunningTime="2026-11-02 18:30:03.530378411 +0000 UTC m=+4034.370984345"
Nov 02 18:30:03.769837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.769107    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" podStartSLOduration=3.769088599 podStartE2EDuration="3.769088599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.760025806 +0000 UTC m=+4034.600631742" watchObservedRunningTime="2026-11-02 18:30:03.769088599 +0000 UTC m=+4034.609694536"
Nov 02 18:30:03.881021 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.880960    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" podStartSLOduration=3.880944229 podStartE2EDuration="3.880944229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.878449128 +0000 UTC m=+4034.719055063" watchObservedRunningTime="2026-11-02 18:30:03.880944229 +0000 UTC m=+4034.721550162"
Nov 02 18:30:03.981776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:03.981547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:04.318472 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.318377    4476 generic.go:356] "Generic (PLEG): container finished" podID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerID="a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea" exitCode=60
Nov 02 18:30:04.318472 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.318456    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" event={"ID":"2f8196cd-5d00-4e27-8c1c-a217bc800de9","Type":"ContainerDied","Data":"a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea"}
Nov 02 18:30:04.319066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.318546    4476 scope.go:122] "RemoveContainer" containerID="1cbfef6b47b789fd50bf5ad76c581595abf4aa3ea3e0a7b38cb5b0520b4d4251"
Nov 02 18:30:04.319066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.318737    4476 scope.go:122] "RemoveContainer" containerID="a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea"
Nov 02 18:30:04.319152 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:04.319062    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn_odh-ai-gateway-infra(2f8196cd-5d00-4e27-8c1c-a217bc800de9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9"
Nov 02 18:30:04.320579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.320560    4476 generic.go:356] "Generic (PLEG): container finished" podID="f74db920-62da-484a-8190-1eb3f793166e" containerID="aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70" exitCode=60
Nov 02 18:30:04.320724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.320669    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" event={"ID":"f74db920-62da-484a-8190-1eb3f793166e","Type":"ContainerDied","Data":"aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70"}
Nov 02 18:30:04.320908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.320889    4476 scope.go:122] "RemoveContainer" containerID="aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70"
Nov 02 18:30:04.321195 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:04.321173    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c_odh-ai-gateway-infra(f74db920-62da-484a-8190-1eb3f793166e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" podUID="f74db920-62da-484a-8190-1eb3f793166e"
Nov 02 18:30:04.322744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.322721    4476 generic.go:356] "Generic (PLEG): container finished" podID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerID="c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79" exitCode=60
Nov 02 18:30:04.322854 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.322800    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerDied","Data":"c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79"}
Nov 02 18:30:04.322982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.322969    4476 scope.go:122] "RemoveContainer" containerID="c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79"
Nov 02 18:30:04.323181 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:04.323159    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-scm9f_odh-ai-gateway-infra(07a6f3a3-a72b-4705-9087-4211859f82a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0"
Nov 02 18:30:04.325014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.324993    4476 generic.go:356] "Generic (PLEG): container finished" podID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerID="a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2" exitCode=60
Nov 02 18:30:04.325114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.325071    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerDied","Data":"a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2"}
Nov 02 18:30:04.325229 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.325213    4476 scope.go:122] "RemoveContainer" containerID="a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2"
Nov 02 18:30:04.325424 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:04.325405    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg_odh-ai-gateway-infra(67d9eb84-501f-4f3a-b68a-ae84f6aa86b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4"
Nov 02 18:30:04.327140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.327113    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerStarted","Data":"2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85"}
Nov 02 18:30:04.328754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.328736    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerStarted","Data":"a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2"}
Nov 02 18:30:04.333153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.333137    4476 scope.go:122] "RemoveContainer" containerID="9c33c0d8894b71ddb147ca11847f3599a5110e68c6f729a467b7804d44321700"
Nov 02 18:30:04.377612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.377590    4476 scope.go:122] "RemoveContainer" containerID="3535694c63ca890cdb0e68c0de936e4aab41b6932b68491fe73cbcaccdd321b9"
Nov 02 18:30:04.391566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:04.391547    4476 scope.go:122] "RemoveContainer" containerID="1eb4333a4668f5d3dc02b6c236cc3f5b1bdc6c71f5ea65d88948e2ef2265bcd7"
Nov 02 18:30:05.337558 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.337526    4476 scope.go:122] "RemoveContainer" containerID="a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2"
Nov 02 18:30:05.338104 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:05.337886    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg_odh-ai-gateway-infra(67d9eb84-501f-4f3a-b68a-ae84f6aa86b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4"
Nov 02 18:30:05.350658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.347313    4476 generic.go:356] "Generic (PLEG): container finished" podID="f007f79b-bbf9-4571-af63-eee3b962e409" containerID="2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85" exitCode=60
Nov 02 18:30:05.350658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.347415    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerDied","Data":"2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85"}
Nov 02 18:30:05.350658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.347469    4476 scope.go:122] "RemoveContainer" containerID="71ce6e82127a18b7dcdbeaee40c0a46ddbec8a11fb52d7454ca68c46d6463d70"
Nov 02 18:30:05.350658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.347902    4476 scope.go:122] "RemoveContainer" containerID="2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85"
Nov 02 18:30:05.350658 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:05.348239    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln_odh-ai-gateway-infra(f007f79b-bbf9-4571-af63-eee3b962e409)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" podUID="f007f79b-bbf9-4571-af63-eee3b962e409"
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.360547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.360972    4476 generic.go:356] "Generic (PLEG): container finished" podID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerID="a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2" exitCode=60
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.361102    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerDied","Data":"a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2"}
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.361482    4476 scope.go:122] "RemoveContainer" containerID="a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2"
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:05.361823    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw_odh-ai-gateway-infra(5cd833b6-aeb3-45ca-a6a4-cf2bdab12827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827"
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.364198    4476 scope.go:122] "RemoveContainer" containerID="a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea"
Nov 02 18:30:05.365655 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:05.364460    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn_odh-ai-gateway-infra(2f8196cd-5d00-4e27-8c1c-a217bc800de9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9"
Nov 02 18:30:05.370533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.370372    4476 scope.go:122] "RemoveContainer" containerID="aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70"
Nov 02 18:30:05.370703 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:05.370677    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c_odh-ai-gateway-infra(f74db920-62da-484a-8190-1eb3f793166e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" podUID="f74db920-62da-484a-8190-1eb3f793166e"
Nov 02 18:30:05.372181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.372163    4476 scope.go:122] "RemoveContainer" containerID="c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79"
Nov 02 18:30:05.372347 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:05.372327    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-scm9f_odh-ai-gateway-infra(07a6f3a3-a72b-4705-9087-4211859f82a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0"
Nov 02 18:30:05.376497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.376473    4476 scope.go:122] "RemoveContainer" containerID="1113c1f837cbd6c8b54a922921bc48115ff45462b0d5aab0f1b386aba270a48e"
Nov 02 18:30:05.565585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:05.565398    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:06.383004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:06.382976    4476 scope.go:122] "RemoveContainer" containerID="2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85"
Nov 02 18:30:06.383398 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:06.383235    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln_odh-ai-gateway-infra(f007f79b-bbf9-4571-af63-eee3b962e409)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" podUID="f007f79b-bbf9-4571-af63-eee3b962e409"
Nov 02 18:30:06.384737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:06.384718    4476 scope.go:122] "RemoveContainer" containerID="a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2"
Nov 02 18:30:06.384936 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:06.384915    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw_odh-ai-gateway-infra(5cd833b6-aeb3-45ca-a6a4-cf2bdab12827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827"
Nov 02 18:30:06.534395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:06.534358    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:06.941418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:06.941373    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:07.545514 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:07.545457    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:08.049357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:08.049317    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:09.188200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:09.188163    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:10.419690 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:10.419659    4476 generic.go:356] "Generic (PLEG): container finished" podID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerID="273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26" exitCode=0
Nov 02 18:30:10.420122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:10.419724    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" event={"ID":"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9","Type":"ContainerDied","Data":"273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"}
Nov 02 18:30:10.420166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:10.420118    4476 scope.go:122] "RemoveContainer" containerID="273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"
Nov 02 18:30:10.421692 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:10.421667    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:30:10.421752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:10.421733    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:30:10.421788 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:10.421765    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:30:11.444653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:11.442186    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" event={"ID":"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9","Type":"ContainerStarted","Data":"8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0"}
Nov 02 18:30:11.511304 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:11.511272    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:30:11.512449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:11.512426    4476 patch_prober.go:29] interesting pod/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 02 18:30:11.512530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:11.512474    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 02 18:30:12.199463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:12.199430    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:12.408212 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:12.408169    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:12.511043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:12.510965    4476 patch_prober.go:29] interesting pod/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 02 18:30:12.511043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:12.511016    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 02 18:30:12.606268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:12.606224    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:13.511527 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:13.511488    4476 patch_prober.go:29] interesting pod/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 02 18:30:13.511955 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:13.511568    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 02 18:30:14.329702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:14.329667    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:30:14.375968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:14.375938    4476 scope.go:122] "RemoveContainer" containerID="a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea"
Nov 02 18:30:14.403079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:14.403057    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:14.511253 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:14.511220    4476 patch_prober.go:29] interesting pod/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 02 18:30:14.511432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:14.511281    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 02 18:30:15.475660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.475427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:15.510879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.510844    4476 patch_prober.go:29] interesting pod/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 02 18:30:15.511070 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.510896    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 02 18:30:15.511359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.511336    4476 generic.go:356] "Generic (PLEG): container finished" podID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerID="c97aa741c78b8e69dbc3e26ecb290e1eaef5d383e84e33333663b7f64dab5751" exitCode=60
Nov 02 18:30:15.511439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.511410    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" event={"ID":"2f8196cd-5d00-4e27-8c1c-a217bc800de9","Type":"ContainerDied","Data":"c97aa741c78b8e69dbc3e26ecb290e1eaef5d383e84e33333663b7f64dab5751"}
Nov 02 18:30:15.511511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.511497    4476 scope.go:122] "RemoveContainer" containerID="a3826444ef71801dbeab77650822ec9cd637597e9c262c772a5eb1f3fb8dedea"
Nov 02 18:30:15.511948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.511932    4476 scope.go:122] "RemoveContainer" containerID="c97aa741c78b8e69dbc3e26ecb290e1eaef5d383e84e33333663b7f64dab5751"
Nov 02 18:30:15.512345 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:15.512233    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn_odh-ai-gateway-infra(2f8196cd-5d00-4e27-8c1c-a217bc800de9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9"
Nov 02 18:30:15.516931 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:15.516820    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8196cd_5d00_4e27_8c1c_a217bc800de9.slice/crio-conmon-c97aa741c78b8e69dbc3e26ecb290e1eaef5d383e84e33333663b7f64dab5751.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:30:15.678177 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.678086    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:15.834208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.834135    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"]
Nov 02 18:30:15.857652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.851932    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:15.925652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.909500    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6cf852\""
Nov 02 18:30:15.925652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.910103    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6cf852-dockercfg-22g9l\""
Nov 02 18:30:15.931649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.926355    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:15.931649 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.926449    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kwd\" (UniqueName: \"kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd\") pod \"payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:15.962364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:15.962321    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"]
Nov 02 18:30:16.028264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.028219    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:16.036861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.036816    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kwd\" (UniqueName: \"kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd\") pod \"payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:16.062038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.062004    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:16.095699 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.095520    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kwd\" (UniqueName: \"kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd\") pod \"payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:16.181867 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.181048    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:16.376220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.376179    4476 scope.go:122] "RemoveContainer" containerID="a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2"
Nov 02 18:30:16.376564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.376542    4476 scope.go:122] "RemoveContainer" containerID="aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70"
Nov 02 18:30:16.511236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.511196    4476 patch_prober.go:29] interesting pod/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 02 18:30:16.511749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.511254    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 02 18:30:16.593536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.593502    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"]
Nov 02 18:30:16.653586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.653536    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"]
Nov 02 18:30:16.826588 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.826566    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:16.885741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.885657    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca\") pod \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") "
Nov 02 18:30:16.885910 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.885813    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z\" (UniqueName: \"kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z\") pod \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\" (UID: \"2f8196cd-5d00-4e27-8c1c-a217bc800de9\") "
Nov 02 18:30:16.886038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.885989    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca" pod "2f8196cd-5d00-4e27-8c1c-a217bc800de9" (UID: "2f8196cd-5d00-4e27-8c1c-a217bc800de9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:16.886281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.886262    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8196cd-5d00-4e27-8c1c-a217bc800de9-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:16.892823 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.892789    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z" pod "2f8196cd-5d00-4e27-8c1c-a217bc800de9" (UID: "2f8196cd-5d00-4e27-8c1c-a217bc800de9"). InnerVolumeSpecName "kube-api-access-87w8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:16.988674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:16.987031    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87w8z\" (UniqueName: \"kubernetes.io/projected/2f8196cd-5d00-4e27-8c1c-a217bc800de9-kube-api-access-87w8z\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:17.376010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.375985    4476 scope.go:122] "RemoveContainer" containerID="c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79"
Nov 02 18:30:17.516079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.516050    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:30:17.517324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.517303    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:30:17.531748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.531723    4476 generic.go:356] "Generic (PLEG): container finished" podID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerID="b8f33ed4799f3f53d83ff275d12afe3336753e17881ecbca493987f7c04002b1" exitCode=60
Nov 02 18:30:17.531870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.531798    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerDied","Data":"b8f33ed4799f3f53d83ff275d12afe3336753e17881ecbca493987f7c04002b1"}
Nov 02 18:30:17.531870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.531845    4476 scope.go:122] "RemoveContainer" containerID="a13d2eabd2b6a1be82cb4811c21077fcd154fbd81b617bad95fdd85f6527f3b2"
Nov 02 18:30:17.532259 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.532244    4476 scope.go:122] "RemoveContainer" containerID="b8f33ed4799f3f53d83ff275d12afe3336753e17881ecbca493987f7c04002b1"
Nov 02 18:30:17.532556 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:17.532527    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg_odh-ai-gateway-infra(67d9eb84-501f-4f3a-b68a-ae84f6aa86b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4"
Nov 02 18:30:17.533739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.533717    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg" event={"ID":"38735db1-7d04-4711-9dab-a0f55472982e","Type":"ContainerStarted","Data":"a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a"}
Nov 02 18:30:17.533841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.533760    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg" event={"ID":"38735db1-7d04-4711-9dab-a0f55472982e","Type":"ContainerStarted","Data":"c49c4c3fa4e46f1f900a6c494360102c567eeae4c4e6e0dcc47ba33ce366f3b2"}
Nov 02 18:30:17.533841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.533786    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:17.535022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.535004    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"
Nov 02 18:30:17.535247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.535037    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn" event={"ID":"2f8196cd-5d00-4e27-8c1c-a217bc800de9","Type":"ContainerDied","Data":"0b56dcaff5c1a5fa86158d4eba864bceba6be1b9cbcb562b548e851a0a0dd4c1"}
Nov 02 18:30:17.537051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.537033    4476 generic.go:356] "Generic (PLEG): container finished" podID="f74db920-62da-484a-8190-1eb3f793166e" containerID="a3be4fece4b92616cf9c86df6f1dfef723f72fbc37d32ccb8f5e11aab6b8869a" exitCode=60
Nov 02 18:30:17.537154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.537100    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" event={"ID":"f74db920-62da-484a-8190-1eb3f793166e","Type":"ContainerDied","Data":"a3be4fece4b92616cf9c86df6f1dfef723f72fbc37d32ccb8f5e11aab6b8869a"}
Nov 02 18:30:17.537537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.537519    4476 scope.go:122] "RemoveContainer" containerID="a3be4fece4b92616cf9c86df6f1dfef723f72fbc37d32ccb8f5e11aab6b8869a"
Nov 02 18:30:17.537835 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:17.537810    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c_odh-ai-gateway-infra(f74db920-62da-484a-8190-1eb3f793166e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" podUID="f74db920-62da-484a-8190-1eb3f793166e"
Nov 02 18:30:17.560189 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.560166    4476 scope.go:122] "RemoveContainer" containerID="c97aa741c78b8e69dbc3e26ecb290e1eaef5d383e84e33333663b7f64dab5751"
Nov 02 18:30:17.578757 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.578737    4476 scope.go:122] "RemoveContainer" containerID="aa65dae3382c64c57b3d240445b8b11894d2b7b4660c34bbfec4303b0ea0eb70"
Nov 02 18:30:17.752214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.752159    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg" podStartSLOduration=2.752139273 podStartE2EDuration="2.752139273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:17.746692125 +0000 UTC m=+4048.587298060" watchObservedRunningTime="2026-11-02 18:30:17.752139273 +0000 UTC m=+4048.592745209"
Nov 02 18:30:17.860802 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.860769    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"]
Nov 02 18:30:17.865100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:17.865071    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894070-h4ldn"]
Nov 02 18:30:18.376508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:18.376478    4476 scope.go:122] "RemoveContainer" containerID="a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2"
Nov 02 18:30:18.376829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:18.376802    4476 scope.go:122] "RemoveContainer" containerID="2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85"
Nov 02 18:30:18.553036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:18.551323    4476 generic.go:356] "Generic (PLEG): container finished" podID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerID="f04bd4ada07389ac5ecedbaa03c4e1722cdac5d59f4046e4786ee1cd17d16ca6" exitCode=60
Nov 02 18:30:18.553036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:18.551481    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerDied","Data":"f04bd4ada07389ac5ecedbaa03c4e1722cdac5d59f4046e4786ee1cd17d16ca6"}
Nov 02 18:30:18.553036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:18.551537    4476 scope.go:122] "RemoveContainer" containerID="c9fd3808d2b9fc2c4d55e1215cb51f996e9c8568e332ccee1378dd1fc996fc79"
Nov 02 18:30:18.553036 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:18.552009    4476 scope.go:122] "RemoveContainer" containerID="f04bd4ada07389ac5ecedbaa03c4e1722cdac5d59f4046e4786ee1cd17d16ca6"
Nov 02 18:30:18.553036 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:18.552347    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-scm9f_odh-ai-gateway-infra(07a6f3a3-a72b-4705-9087-4211859f82a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0"
Nov 02 18:30:19.104325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.104290    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"]
Nov 02 18:30:19.220208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.220144    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"]
Nov 02 18:30:19.387976 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.387939    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" path="/var/lib/kubelet/pods/2f8196cd-5d00-4e27-8c1c-a217bc800de9/volumes"
Nov 02 18:30:19.400462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.400233    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:19.403871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.403852    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:19.495319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.495284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:19.519597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.519565    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g\" (UniqueName: \"kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g\") pod \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") "
Nov 02 18:30:19.528015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.527982    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4\" (UniqueName: \"kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4\") pod \"f74db920-62da-484a-8190-1eb3f793166e\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") "
Nov 02 18:30:19.528199 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.528076    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca\") pod \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\" (UID: \"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4\") "
Nov 02 18:30:19.528199 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.528147    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca\") pod \"f74db920-62da-484a-8190-1eb3f793166e\" (UID: \"f74db920-62da-484a-8190-1eb3f793166e\") "
Nov 02 18:30:19.528820 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.528786    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca" pod "67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" (UID: "67d9eb84-501f-4f3a-b68a-ae84f6aa86b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:19.529195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.529169    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca" pod "f74db920-62da-484a-8190-1eb3f793166e" (UID: "f74db920-62da-484a-8190-1eb3f793166e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:19.529945 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.529923    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g" pod "67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" (UID: "67d9eb84-501f-4f3a-b68a-ae84f6aa86b4"). InnerVolumeSpecName "kube-api-access-hcb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:19.533808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.533781    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4" pod "f74db920-62da-484a-8190-1eb3f793166e" (UID: "f74db920-62da-484a-8190-1eb3f793166e"). InnerVolumeSpecName "kube-api-access-2x9j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:19.573586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.573556    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"
Nov 02 18:30:19.573586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.573569    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg" event={"ID":"67d9eb84-501f-4f3a-b68a-ae84f6aa86b4","Type":"ContainerDied","Data":"5989c5c82c1e600523e2a35e0411fadfbb474f0e32be527ba0b681a128034417"}
Nov 02 18:30:19.574172 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.573676    4476 scope.go:122] "RemoveContainer" containerID="b8f33ed4799f3f53d83ff275d12afe3336753e17881ecbca493987f7c04002b1"
Nov 02 18:30:19.578014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.577985    4476 generic.go:356] "Generic (PLEG): container finished" podID="f007f79b-bbf9-4571-af63-eee3b962e409" containerID="b52e33c01a08b7441a5c28dd5c5eed327988eca9d93cb6b2fe766d2b1df9a780" exitCode=60
Nov 02 18:30:19.578180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.578115    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerDied","Data":"b52e33c01a08b7441a5c28dd5c5eed327988eca9d93cb6b2fe766d2b1df9a780"}
Nov 02 18:30:19.578575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.578556    4476 scope.go:122] "RemoveContainer" containerID="b52e33c01a08b7441a5c28dd5c5eed327988eca9d93cb6b2fe766d2b1df9a780"
Nov 02 18:30:19.578971 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:19.578945    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln_odh-ai-gateway-infra(f007f79b-bbf9-4571-af63-eee3b962e409)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" podUID="f007f79b-bbf9-4571-af63-eee3b962e409"
Nov 02 18:30:19.582391 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.582364    4476 generic.go:356] "Generic (PLEG): container finished" podID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerID="46ee7e7f4b36f9c30fc599fa4957f65f1ee90c5419796232b092d9d7e3df3a13" exitCode=60
Nov 02 18:30:19.582550 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.582533    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerDied","Data":"46ee7e7f4b36f9c30fc599fa4957f65f1ee90c5419796232b092d9d7e3df3a13"}
Nov 02 18:30:19.582991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.582972    4476 scope.go:122] "RemoveContainer" containerID="46ee7e7f4b36f9c30fc599fa4957f65f1ee90c5419796232b092d9d7e3df3a13"
Nov 02 18:30:19.583316 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:19.583295    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw_odh-ai-gateway-infra(5cd833b6-aeb3-45ca-a6a4-cf2bdab12827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827"
Nov 02 18:30:19.585618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.585592    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c" event={"ID":"f74db920-62da-484a-8190-1eb3f793166e","Type":"ContainerDied","Data":"676ae0666aba02744b7ab800ce5ce2d4df5506c1f627bef140c3e62863dcd1dd"}
Nov 02 18:30:19.585755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.585737    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"
Nov 02 18:30:19.589038 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.589014    4476 scope.go:122] "RemoveContainer" containerID="2bf91f52213638ae105f7aa421497ef357ac619db464f3b0b9d21ae33ecf7d85"
Nov 02 18:30:19.600442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.600423    4476 scope.go:122] "RemoveContainer" containerID="a1d2c81f2a7dcf69d2602723f83f467f6867af854dff6da93ee77c4483acd7a2"
Nov 02 18:30:19.611710 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.611692    4476 scope.go:122] "RemoveContainer" containerID="a3be4fece4b92616cf9c86df6f1dfef723f72fbc37d32ccb8f5e11aab6b8869a"
Nov 02 18:30:19.630207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.630157    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcb8g\" (UniqueName: \"kubernetes.io/projected/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-kube-api-access-hcb8g\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.630207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.630183    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9j4\" (UniqueName: \"kubernetes.io/projected/f74db920-62da-484a-8190-1eb3f793166e-kube-api-access-2x9j4\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.630207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.630194    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.630207 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.630205    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74db920-62da-484a-8190-1eb3f793166e-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.686309 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.686271    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"]
Nov 02 18:30:19.766667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.763349    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"]
Nov 02 18:30:19.770213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.770170    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894070-c2j2c"]
Nov 02 18:30:19.819134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.819108    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:19.933870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.933786    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp\" (UniqueName: \"kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp\") pod \"07a6f3a3-a72b-4705-9087-4211859f82a0\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") "
Nov 02 18:30:19.934029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.933952    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca\") pod \"07a6f3a3-a72b-4705-9087-4211859f82a0\" (UID: \"07a6f3a3-a72b-4705-9087-4211859f82a0\") "
Nov 02 18:30:19.934287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.934256    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca" pod "07a6f3a3-a72b-4705-9087-4211859f82a0" (UID: "07a6f3a3-a72b-4705-9087-4211859f82a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:19.936063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.936036    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp" pod "07a6f3a3-a72b-4705-9087-4211859f82a0" (UID: "07a6f3a3-a72b-4705-9087-4211859f82a0"). InnerVolumeSpecName "kube-api-access-smrzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:19.998314 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:19.998277    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"]
Nov 02 18:30:20.004569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.004541    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894070-qb2cg"]
Nov 02 18:30:20.035505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.035473    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smrzp\" (UniqueName: \"kubernetes.io/projected/07a6f3a3-a72b-4705-9087-4211859f82a0-kube-api-access-smrzp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.035505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.035499    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a6f3a3-a72b-4705-9087-4211859f82a0-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.603238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.603158    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f" event={"ID":"07a6f3a3-a72b-4705-9087-4211859f82a0","Type":"ContainerDied","Data":"992a04932d7fd4dd78e409bd47d0bad4400713dcaa65d4cae6ee5e7adcd2e443"}
Nov 02 18:30:20.603238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.603180    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"
Nov 02 18:30:20.603238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.603223    4476 scope.go:122] "RemoveContainer" containerID="f04bd4ada07389ac5ecedbaa03c4e1722cdac5d59f4046e4786ee1cd17d16ca6"
Nov 02 18:30:20.770351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.770317    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"]
Nov 02 18:30:20.775091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.775064    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-scm9f"]
Nov 02 18:30:20.895934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:20.895896    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"]
Nov 02 18:30:21.049874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.049850    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:21.140656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.128117    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"]
Nov 02 18:30:21.152756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.148243    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf\" (UniqueName: \"kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf\") pod \"f007f79b-bbf9-4571-af63-eee3b962e409\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") "
Nov 02 18:30:21.152756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.148909    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca\") pod \"f007f79b-bbf9-4571-af63-eee3b962e409\" (UID: \"f007f79b-bbf9-4571-af63-eee3b962e409\") "
Nov 02 18:30:21.152756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.149247    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca" pod "f007f79b-bbf9-4571-af63-eee3b962e409" (UID: "f007f79b-bbf9-4571-af63-eee3b962e409"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:21.152756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.149874    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f007f79b-bbf9-4571-af63-eee3b962e409-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:21.155466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.155429    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf" pod "f007f79b-bbf9-4571-af63-eee3b962e409" (UID: "f007f79b-bbf9-4571-af63-eee3b962e409"). InnerVolumeSpecName "kube-api-access-mnrpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:21.250797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.250762    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnrpf\" (UniqueName: \"kubernetes.io/projected/f007f79b-bbf9-4571-af63-eee3b962e409-kube-api-access-mnrpf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:21.337444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.337415    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:21.384974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.384927    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" path="/var/lib/kubelet/pods/07a6f3a3-a72b-4705-9087-4211859f82a0/volumes"
Nov 02 18:30:21.385421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.385405    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" path="/var/lib/kubelet/pods/67d9eb84-501f-4f3a-b68a-ae84f6aa86b4/volumes"
Nov 02 18:30:21.385853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.385837    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74db920-62da-484a-8190-1eb3f793166e" path="/var/lib/kubelet/pods/f74db920-62da-484a-8190-1eb3f793166e/volumes"
Nov 02 18:30:21.452293 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.452197    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca\") pod \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") "
Nov 02 18:30:21.452293 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.452286    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw\" (UniqueName: \"kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw\") pod \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\" (UID: \"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827\") "
Nov 02 18:30:21.452680 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.452649    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca" pod "5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" (UID: "5cd833b6-aeb3-45ca-a6a4-cf2bdab12827"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:21.452995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.452976    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:21.467763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.455395    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw" pod "5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" (UID: "5cd833b6-aeb3-45ca-a6a4-cf2bdab12827"). InnerVolumeSpecName "kube-api-access-btfvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:21.554181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.554150    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btfvw\" (UniqueName: \"kubernetes.io/projected/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827-kube-api-access-btfvw\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:30:21.613466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.613435    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"
Nov 02 18:30:21.613466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.613446    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln" event={"ID":"f007f79b-bbf9-4571-af63-eee3b962e409","Type":"ContainerDied","Data":"0516d411a68d4454d234a7a620f6bad56aa7bebbdbe7f46298fc0a273a282865"}
Nov 02 18:30:21.614081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.613502    4476 scope.go:122] "RemoveContainer" containerID="b52e33c01a08b7441a5c28dd5c5eed327988eca9d93cb6b2fe766d2b1df9a780"
Nov 02 18:30:21.615467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.615394    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"
Nov 02 18:30:21.615612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.615514    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw" event={"ID":"5cd833b6-aeb3-45ca-a6a4-cf2bdab12827","Type":"ContainerDied","Data":"69293040af866bed3f2ba1ca28c7e7c0d26c70623c5ebce69aa71a7a1e40fc4a"}
Nov 02 18:30:21.624428 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.624406    4476 scope.go:122] "RemoveContainer" containerID="46ee7e7f4b36f9c30fc599fa4957f65f1ee90c5419796232b092d9d7e3df3a13"
Nov 02 18:30:21.781067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.780986    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"]
Nov 02 18:30:21.786522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.786496    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894070-77hln"]
Nov 02 18:30:21.891225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.891192    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"]
Nov 02 18:30:21.896781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:21.896755    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-662a80-29894070-88crw"]
Nov 02 18:30:23.383721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:23.383688    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" path="/var/lib/kubelet/pods/5cd833b6-aeb3-45ca-a6a4-cf2bdab12827/volumes"
Nov 02 18:30:23.384110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:23.384093    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" path="/var/lib/kubelet/pods/f007f79b-bbf9-4571-af63-eee3b962e409/volumes"
Nov 02 18:30:24.283195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:24.283164    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:28.565362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:28.565326    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:30:29.060108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:29.060072    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:29.061945 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:29.061916    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:30:32.719283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:32.719244    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:33.194521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:33.194472    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:33.795362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:33.795264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:34.327102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:34.327062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:34.532546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:34.532504    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:37.931946 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:37.931914    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:38.209639 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:38.209533    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:39.313027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:39.312989    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:41.171182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.171147    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:41.354774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.354728    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:41.542261 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542185    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"]
Nov 02 18:30:41.542769 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542755    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542770    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542782    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542787    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542799    4476 cpu_manager.go:414] "containerMap: removing container" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542803    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542812    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542819    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542826    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542831    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.542835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542836    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542840    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542851    4476 cpu_manager.go:414] "containerMap: removing container" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542855    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542868    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542872    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542881    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542886    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542895    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542898    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542906    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542909    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542917    4476 cpu_manager.go:414] "containerMap: removing container" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542922    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542932    4476 cpu_manager.go:414] "containerMap: removing container" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542936    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542943    4476 cpu_manager.go:414] "containerMap: removing container" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542947    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542955    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542959    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542965    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.542969    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543043    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543053    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543059    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543064    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543069    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543073    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543078    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543083    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543090    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543096    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8196cd-5d00-4e27-8c1c-a217bc800de9" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543102    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543106    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f007f79b-bbf9-4571-af63-eee3b962e409" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543115    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.543156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.543120    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.546177 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.546157    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.597017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.596986    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.597188 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.597035    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxnc\" (UniqueName: \"kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc\") pod \"payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.602162 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.602136    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"]
Nov 02 18:30:41.669453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.668004    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a51e27\""
Nov 02 18:30:41.669453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.668157    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a51e27-dockercfg-dvkpc\""
Nov 02 18:30:41.701940 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.701905    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.702157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.701958    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxnc\" (UniqueName: \"kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc\") pod \"payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.703024 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.702998    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.721087 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721053    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"]
Nov 02 18:30:41.721612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721597    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.721676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721614    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.721676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721653    4476 cpu_manager.go:414] "containerMap: removing container" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.721676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721659    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.721777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721767    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d9eb84-501f-4f3a-b68a-ae84f6aa86b4" containerName="cleanup"
Nov 02 18:30:41.721812 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721777    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd833b6-aeb3-45ca-a6a4-cf2bdab12827" containerName="cleanup"
Nov 02 18:30:41.721812 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721786    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74db920-62da-484a-8190-1eb3f793166e" containerName="cleanup"
Nov 02 18:30:41.721812 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.721792    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a6f3a3-a72b-4705-9087-4211859f82a0" containerName="cleanup"
Nov 02 18:30:41.725523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.725501    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:41.753492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.753445    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"]
Nov 02 18:30:41.758456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.758435    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxnc\" (UniqueName: \"kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc\") pod \"payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.803248 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.803181    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:41.803248 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.803229    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8k2\" (UniqueName: \"kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2\") pod \"payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:41.859904 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.859870    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:41.905100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.905065    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:41.905285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.905141    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8k2\" (UniqueName: \"kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2\") pod \"payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:41.905890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.905868    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:41.961942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:41.961912    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8k2\" (UniqueName: \"kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2\") pod \"payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:42.040009 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.039975    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:42.242357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.242327    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"]
Nov 02 18:30:42.243586 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:30:42.243548    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca107b84_5b8c_4781_bab0_49215ab7ccbf.slice/crio-82f31e68e1305113e13dc3ab9698393070a1cb33359b68ce34969aba16e23520 WatchSource:0}: Error finding container 82f31e68e1305113e13dc3ab9698393070a1cb33359b68ce34969aba16e23520: Status 404 returned error can't find the container with id 82f31e68e1305113e13dc3ab9698393070a1cb33359b68ce34969aba16e23520
Nov 02 18:30:42.422035 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:30:42.422006    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aec8aaf_241f_43bb_8e6b_afa034a9da2c.slice/crio-a8f46abe3439d84411c46b27915a0b0c59823d03e700f420ed8c51917988651f WatchSource:0}: Error finding container a8f46abe3439d84411c46b27915a0b0c59823d03e700f420ed8c51917988651f: Status 404 returned error can't find the container with id a8f46abe3439d84411c46b27915a0b0c59823d03e700f420ed8c51917988651f
Nov 02 18:30:42.434517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.434489    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"]
Nov 02 18:30:42.831518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.831484    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9" event={"ID":"4aec8aaf-241f-43bb-8e6b-afa034a9da2c","Type":"ContainerStarted","Data":"f840d233e8ef4c593fbf8fc6ce78eb55bd7724ce45b5f1c4c96861795a64c343"}
Nov 02 18:30:42.831737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.831534    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9" event={"ID":"4aec8aaf-241f-43bb-8e6b-afa034a9da2c","Type":"ContainerStarted","Data":"a8f46abe3439d84411c46b27915a0b0c59823d03e700f420ed8c51917988651f"}
Nov 02 18:30:42.831737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.831583    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:42.833333 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.833311    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj" event={"ID":"ca107b84-5b8c-4781-bab0-49215ab7ccbf","Type":"ContainerStarted","Data":"15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323"}
Nov 02 18:30:42.833459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.833350    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj" event={"ID":"ca107b84-5b8c-4781-bab0-49215ab7ccbf","Type":"ContainerStarted","Data":"82f31e68e1305113e13dc3ab9698393070a1cb33359b68ce34969aba16e23520"}
Nov 02 18:30:42.833459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.833411    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:42.948075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:42.948015    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9" podStartSLOduration=1.947994274 podStartE2EDuration="1.947994274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:42.944924703 +0000 UTC m=+4073.785530638" watchObservedRunningTime="2026-11-02 18:30:42.947994274 +0000 UTC m=+4073.788600210"
Nov 02 18:30:43.077969 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:43.077874    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj" podStartSLOduration=2.077852943 podStartE2EDuration="2.077852943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:43.073227771 +0000 UTC m=+4073.913833704" watchObservedRunningTime="2026-11-02 18:30:43.077852943 +0000 UTC m=+4073.918458880"
Nov 02 18:30:43.969993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:43.969959    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"]
Nov 02 18:30:43.974246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:43.974229    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.031466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.031436    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.031642 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.031507    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkjv\" (UniqueName: \"kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.031702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.031677    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.031759 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.031744    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.075762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.075711    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c70712-serving-cert\""
Nov 02 18:30:44.079411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.079384    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"]
Nov 02 18:30:44.132954 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.132923    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.133153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.132982    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.133153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.133024    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkjv\" (UniqueName: \"kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.133153 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:44.133071    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-serving-cert: secret "maas-api-e2e-inf-a-c70712-serving-cert" not found
Nov 02 18:30:44.133153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.133102    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.133153 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:44.133150    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls podName:666f4497-aec3-4f56-9369-322b6adf8747 nodeName:}" failed. No retries permitted until 2026-11-02 18:30:44.633128461 +0000 UTC m=+4075.473734375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls") pod "maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" (UID: "666f4497-aec3-4f56-9369-322b6adf8747") : secret "maas-api-e2e-inf-a-c70712-serving-cert" not found
Nov 02 18:30:44.133726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.133704    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.135874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.135786    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.190962 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.190931    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkjv\" (UniqueName: \"kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.215889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.215855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:44.449810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.449777    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:44.626565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.626512    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:44.638942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.638909    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.641526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.641505    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\") pod \"maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:44.889011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:44.888970    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:45.280786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:45.280749    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"]
Nov 02 18:30:45.859820 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:45.859776    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" event={"ID":"666f4497-aec3-4f56-9369-322b6adf8747","Type":"ContainerStarted","Data":"8098c40bee14102ead4b28ec8569322d2f86061ab089e551c3c68162f7152f15"}
Nov 02 18:30:46.868812 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:46.868775    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" event={"ID":"666f4497-aec3-4f56-9369-322b6adf8747","Type":"ContainerStarted","Data":"7dbb02e1042d3a8db247027e32c63b2749351619de4b1a814cf649661b859705"}
Nov 02 18:30:46.869450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:46.869427    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:46.991959 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:46.991881    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" podStartSLOduration=3.085358821 podStartE2EDuration="3.991859599s" totalImagesPullingTime="906.500778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:46.987087368 +0000 UTC m=+4077.827693306" watchObservedRunningTime="2026-11-02 18:30:46.991859599 +0000 UTC m=+4077.832465538"
Nov 02 18:30:49.077493 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:49.077457    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:52.882606 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:52.882576    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:30:53.844816 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:53.844784    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:30:53.845012 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:53.844870    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:30:53.882561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:53.882523    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:58.575999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:30:58.575964    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:59.059477 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:30:59.059425    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:31:03.065051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:03.064971    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:03.259519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:03.259482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:08.023763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:08.023726    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:08.835588 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:08.835553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:09.290236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:09.290191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:09.935652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:09.925568    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:10.326200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:10.326064    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:13.170019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:13.169979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:14.114824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:14.114787    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:14.318168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:14.318132    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:17.394486 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:17.394450    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:17.585354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:17.585319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:18.761499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:18.761431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:23.604109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:23.604069    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:26.056664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:26.056599    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"]
Nov 02 18:31:26.101618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:26.101587    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b794d09-859h7"]
Nov 02 18:31:27.383794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:27.383761    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f93889-423f-47a0-a1ec-782e3450be57" path="/var/lib/kubelet/pods/31f93889-423f-47a0-a1ec-782e3450be57/volumes"
Nov 02 18:31:28.410289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:28.410252    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:29.063580 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:31:29.063538    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:31:33.171349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:33.171311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:37.911099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:37.911048    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:42.256195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.256157    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"]
Nov 02 18:31:42.260724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.260701    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.312927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.312888    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6cf852-serving-cert\""
Nov 02 18:31:42.316764 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.316734    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"]
Nov 02 18:31:42.333037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.333009    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnt5\" (UniqueName: \"kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.333179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.333066    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.333262 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.333237    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.333320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.333284    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.434522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.434490    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.434522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.434523    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.434785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.434592    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnt5\" (UniqueName: \"kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.434785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.434678    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.434785 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:31:42.434700    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-serving-cert: secret "maas-api-e2e-inf-b-6cf852-serving-cert" not found
Nov 02 18:31:42.434785 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:31:42.434771    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls podName:ce22faf9-02ec-4aa3-a84c-7ad8c9f93827 nodeName:}" failed. No retries permitted until 2026-11-02 18:31:42.934750275 +0000 UTC m=+4133.775356187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls") pod "maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" (UID: "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827") : secret "maas-api-e2e-inf-b-6cf852-serving-cert" not found
Nov 02 18:31:42.435336 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.435318    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.437219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.437200    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.493522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.493490    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnt5\" (UniqueName: \"kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.495180 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.495154    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:42.686057 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.686015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:42.869031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.868992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:42.943548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.943461    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:42.946274 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:42.946242    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:43.176365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:43.176323    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:43.562193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:43.562167    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"]
Nov 02 18:31:43.564429 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:31:43.564400    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce22faf9_02ec_4aa3_a84c_7ad8c9f93827.slice/crio-2978b9f50aae1e2221fe7c5fea066fbc22997bdc9563a64afea0b15f1e4d4517 WatchSource:0}: Error finding container 2978b9f50aae1e2221fe7c5fea066fbc22997bdc9563a64afea0b15f1e4d4517: Status 404 returned error can't find the container with id 2978b9f50aae1e2221fe7c5fea066fbc22997bdc9563a64afea0b15f1e4d4517
Nov 02 18:31:44.338348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:44.338313    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" event={"ID":"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827","Type":"ContainerStarted","Data":"2978b9f50aae1e2221fe7c5fea066fbc22997bdc9563a64afea0b15f1e4d4517"}
Nov 02 18:31:45.347619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:45.347579    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" event={"ID":"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827","Type":"ContainerStarted","Data":"4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82"}
Nov 02 18:31:45.348097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:45.347681    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:45.464620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:45.464569    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" podStartSLOduration=2.6515798310000003 podStartE2EDuration="3.464554712s" totalImagesPullingTime="812.974881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:45.460982359 +0000 UTC m=+4136.301588294" watchObservedRunningTime="2026-11-02 18:31:45.464554712 +0000 UTC m=+4136.305160646"
Nov 02 18:31:47.470447 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:47.470414    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:51.319085 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:51.319049    4476 scope.go:122] "RemoveContainer" containerID="48dea6fc9056ecfd05b28b758c92d12a1429dec500fe205d67700f15e02b19da"
Nov 02 18:31:51.329414 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:51.329393    4476 scope.go:122] "RemoveContainer" containerID="de2f077a63f9c8f97b5bbeaf4a5391cfe9e92cfdadabf5dd03d3a03d10324d35"
Nov 02 18:31:51.363151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:51.363130    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:31:52.398850 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:52.398804    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:53.760974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:53.760936    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:53.928945 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:53.928913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:56.865455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:31:56.865421    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:59.071306 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:31:59.071269    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:32:01.478875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:01.478840    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:03.655366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:03.655320    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:06.151515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:06.151472    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:10.933267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:10.933227    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:11.142430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:11.142397    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:11.352015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:11.351936    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:15.846560 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:15.846529    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:20.610747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:20.610707    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:24.903416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:24.903377    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:25.462835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:25.462780    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:29.059251 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:32:29.059212    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:32:30.290922 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:30.290886    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:30.597950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:30.593257    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:30.776994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:30.776962    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:30.953486 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:30.953449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:35.260565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:35.260256    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:36.589382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:36.589329    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:39.988178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:39.988149    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:44.781278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:44.781118    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:49.332969 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:49.332883    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:49.513277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:49.513245    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:49.688513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:49.688485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:50.110933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:50.110910    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:32:50.118920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:50.118898    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:32:51.364523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:51.364483    4476 scope.go:122] "RemoveContainer" containerID="661c2fd53f9b12e141df289a54b110fcba9d7ab6018945935d0d0eb5ce9dfa5b"
Nov 02 18:32:51.375537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:51.375515    4476 scope.go:122] "RemoveContainer" containerID="938d242ec5a5b65e6747358239f1a77ae21be4ed9f3706252a9fa49438782fdd"
Nov 02 18:32:51.387198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:51.387178    4476 scope.go:122] "RemoveContainer" containerID="23afb7a62eb3c660a830c0ba192e4f08579ef38b9ad78e54ea31f5d73864fc7b"
Nov 02 18:32:54.276477 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:54.276440    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:59.061089 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:32:59.061046    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:32:59.061995 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:32:59.061970    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:00.340571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:00.340535    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:00.518721 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:00.518688    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:03.659051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:03.659015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:03.833519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:03.833484    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:08.313191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:08.313134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:12.972963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:12.972911    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:15.661107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:15.661069    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:17.783008 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:17.782967    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:18.005470 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:18.005437    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:21.364059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:21.364022    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:22.568425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:22.560402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:22.763138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:22.763107    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:26.966154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:26.966114    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:27.867850 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:27.867816    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:29.059810 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:33:29.059765    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:33:32.996642 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:32.996595    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:38.095210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:38.095180    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:41.806533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:41.806483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:43.117409 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:43.117372    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:47.469280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:47.469247    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:47.747846 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:47.747752    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:49.515934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:49.515899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:49.700099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:49.700063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:52.907452 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:52.907409    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:53.078128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:53.078095    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:57.827166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:33:57.827129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:59.064694 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:33:59.064656    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:34:02.062358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:02.062284    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:02.249380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:02.249334    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:03.409363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:03.409327    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:04.100525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:04.100487    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:04.340845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:04.340806    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:05.408453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:05.408416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:05.589284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:05.589237    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:07.534358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:07.534328    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:08.350968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:08.350936    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:09.882071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:09.882023    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:10.075638 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:10.075573    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:13.106442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:13.106412    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:13.284878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:13.284842    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:13.608809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:13.608776    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:18.107561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:18.107528    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:18.680215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:18.680173    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:22.301841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:22.301799    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:34:22.302525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:22.302490    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:34:22.365128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:22.365080    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:34:22.365440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:22.365407    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:34:23.874576 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:23.874541    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:24.141183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:24.141074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:24.322527 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:24.322485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:24.526197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:24.526107    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:29.059781 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:34:29.059741    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:34:29.087763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:29.087736    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:30.110111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:30.110074    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:34.260158 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:34.260093    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:39.409071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:39.409028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:44.460520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:44.460483    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:49.848774 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:49.848725    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:56.108829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:56.108784    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:59.061160 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:34:59.061121    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:34:59.746489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:34:59.746451    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:04.442415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:04.442379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:04.641035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:04.640993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:09.167618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:09.167579    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:13.933162 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:13.933120    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:19.054491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:19.054449    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:21.959749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:21.959715    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:24.036808 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:24.036771    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:24.232505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:24.232476    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:28.880515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:28.880463    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:29.059433 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:35:29.059394    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:35:32.827406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:32.827327    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"]
Nov 02 18:35:32.827852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:32.827550    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" podUID="666f4497-aec3-4f56-9369-322b6adf8747" containerName="maas-api" containerID="cri-o://7dbb02e1042d3a8db247027e32c63b2749351619de4b1a814cf649661b859705" gracePeriod=30
Nov 02 18:35:32.878907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:32.878866    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" podUID="666f4497-aec3-4f56-9369-322b6adf8747" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.224:8443/health\": dial tcp 10.133.0.224:8443: connect: connection refused"
Nov 02 18:35:32.962233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:32.962201    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:33.127231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.127189    4476 generic.go:356] "Generic (PLEG): container finished" podID="666f4497-aec3-4f56-9369-322b6adf8747" containerID="7dbb02e1042d3a8db247027e32c63b2749351619de4b1a814cf649661b859705" exitCode=0
Nov 02 18:35:33.127411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.127267    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" event={"ID":"666f4497-aec3-4f56-9369-322b6adf8747","Type":"ContainerDied","Data":"7dbb02e1042d3a8db247027e32c63b2749351619de4b1a814cf649661b859705"}
Nov 02 18:35:33.127411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.127331    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc" event={"ID":"666f4497-aec3-4f56-9369-322b6adf8747","Type":"ContainerDied","Data":"8098c40bee14102ead4b28ec8569322d2f86061ab089e551c3c68162f7152f15"}
Nov 02 18:35:33.127411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.127356    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8098c40bee14102ead4b28ec8569322d2f86061ab089e551c3c68162f7152f15"
Nov 02 18:35:33.130027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.130000    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:35:33.287235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.287203    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\") pod \"666f4497-aec3-4f56-9369-322b6adf8747\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") "
Nov 02 18:35:33.287446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.287408    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs\") pod \"666f4497-aec3-4f56-9369-322b6adf8747\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") "
Nov 02 18:35:33.287536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.287473    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv\" (UniqueName: \"kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv\") pod \"666f4497-aec3-4f56-9369-322b6adf8747\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") "
Nov 02 18:35:33.287602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.287556    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca\") pod \"666f4497-aec3-4f56-9369-322b6adf8747\" (UID: \"666f4497-aec3-4f56-9369-322b6adf8747\") "
Nov 02 18:35:33.288035 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.287985    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca" pod "666f4497-aec3-4f56-9369-322b6adf8747" (UID: "666f4497-aec3-4f56-9369-322b6adf8747"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:35:33.289714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.289684    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls" pod "666f4497-aec3-4f56-9369-322b6adf8747" (UID: "666f4497-aec3-4f56-9369-322b6adf8747"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:35:33.289804 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.289745    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv" pod "666f4497-aec3-4f56-9369-322b6adf8747" (UID: "666f4497-aec3-4f56-9369-322b6adf8747"). InnerVolumeSpecName "kube-api-access-wrkjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:33.290169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.290149    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs" pod "666f4497-aec3-4f56-9369-322b6adf8747" (UID: "666f4497-aec3-4f56-9369-322b6adf8747"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:35:33.388742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.388710    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/666f4497-aec3-4f56-9369-322b6adf8747-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:33.388742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.388738    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:33.388742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.388749    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/666f4497-aec3-4f56-9369-322b6adf8747-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:33.388986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:33.388758    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkjv\" (UniqueName: \"kubernetes.io/projected/666f4497-aec3-4f56-9369-322b6adf8747-kube-api-access-wrkjv\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:34.134483 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:34.134449    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"
Nov 02 18:35:34.298231 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:34.298201    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"]
Nov 02 18:35:34.347866 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:34.347838    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c70712-7d9df968b5-pt2dc"]
Nov 02 18:35:34.604080 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:34.603976    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:34.780257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:34.780214    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:35.132150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:35.132092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:35.392229 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:35.392139    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666f4497-aec3-4f56-9369-322b6adf8747" path="/var/lib/kubelet/pods/666f4497-aec3-4f56-9369-322b6adf8747/volumes"
Nov 02 18:35:35.840184 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:35.840103    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:36.798845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:36.798811    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:36.983136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:36.983097    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:37.602401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:37.602367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:37.786576 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:37.786540    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:39.830267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:39.830232    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:40.009822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:40.009792    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:44.489115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:44.489067    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:48.746123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.746090    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"]
Nov 02 18:35:48.746893 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.746871    4476 cpu_manager.go:414] "containerMap: removing container" podUID="666f4497-aec3-4f56-9369-322b6adf8747" containerName="maas-api"
Nov 02 18:35:48.746893 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.746893    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666f4497-aec3-4f56-9369-322b6adf8747" containerName="maas-api"
Nov 02 18:35:48.747086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.747074    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="666f4497-aec3-4f56-9369-322b6adf8747" containerName="maas-api"
Nov 02 18:35:48.750731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.750710    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.805471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.805443    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"]
Nov 02 18:35:48.849732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.849703    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:35:48.849897 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.849762    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:35:48.886925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.886897    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.887066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.886936    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlx8r\" (UniqueName: \"kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.887066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.887054    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.989168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.989141    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.989339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.989320    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.989402 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.989351    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlx8r\" (UniqueName: \"kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.989984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.989957    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:48.991703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:48.991684    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:49.047176 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:49.047113    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlx8r\" (UniqueName: \"kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r\") pod \"maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:49.067839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:49.067811    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:49.308466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:49.308326    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:49.450331 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:49.449936    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"]
Nov 02 18:35:49.451475 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:35:49.451438    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89b58d0_fa9b_41cc_8653_cee8298999e0.slice/crio-e068d6a724457244e41f1ee95d9b994b4438c87f7fb87de8b7c5d7dc65c6836d WatchSource:0}: Error finding container e068d6a724457244e41f1ee95d9b994b4438c87f7fb87de8b7c5d7dc65c6836d: Status 404 returned error can't find the container with id e068d6a724457244e41f1ee95d9b994b4438c87f7fb87de8b7c5d7dc65c6836d
Nov 02 18:35:50.247737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:50.247668    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn" event={"ID":"a89b58d0-fa9b-41cc-8653-cee8298999e0","Type":"ContainerStarted","Data":"2af3554ca7ea9cd3fae49b45eace00d0ecbc844b1be45ad412860b6def3a6aea"}
Nov 02 18:35:50.248292 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:50.247745    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn" event={"ID":"a89b58d0-fa9b-41cc-8653-cee8298999e0","Type":"ContainerStarted","Data":"e068d6a724457244e41f1ee95d9b994b4438c87f7fb87de8b7c5d7dc65c6836d"}
Nov 02 18:35:50.363941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:50.363885    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn" podStartSLOduration=2.363870129 podStartE2EDuration="2.363870129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:50.3598025 +0000 UTC m=+4381.200408436" watchObservedRunningTime="2026-11-02 18:35:50.363870129 +0000 UTC m=+4381.204476061"
Nov 02 18:35:51.257998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:51.257962    4476 generic.go:356] "Generic (PLEG): container finished" podID="a89b58d0-fa9b-41cc-8653-cee8298999e0" containerID="2af3554ca7ea9cd3fae49b45eace00d0ecbc844b1be45ad412860b6def3a6aea" exitCode=0
Nov 02 18:35:51.258407 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:51.258053    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn" event={"ID":"a89b58d0-fa9b-41cc-8653-cee8298999e0","Type":"ContainerDied","Data":"2af3554ca7ea9cd3fae49b45eace00d0ecbc844b1be45ad412860b6def3a6aea"}
Nov 02 18:35:52.437679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.437651    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:52.530859 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.530762    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca\") pod \"a89b58d0-fa9b-41cc-8653-cee8298999e0\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") "
Nov 02 18:35:52.531030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.530969    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r\" (UniqueName: \"kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r\") pod \"a89b58d0-fa9b-41cc-8653-cee8298999e0\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") "
Nov 02 18:35:52.531105 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.531043    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca\") pod \"a89b58d0-fa9b-41cc-8653-cee8298999e0\" (UID: \"a89b58d0-fa9b-41cc-8653-cee8298999e0\") "
Nov 02 18:35:52.531186 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.531159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca" pod "a89b58d0-fa9b-41cc-8653-cee8298999e0" (UID: "a89b58d0-fa9b-41cc-8653-cee8298999e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:35:52.531659 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.531621    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:52.533700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.533676    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r" pod "a89b58d0-fa9b-41cc-8653-cee8298999e0" (UID: "a89b58d0-fa9b-41cc-8653-cee8298999e0"). InnerVolumeSpecName "kube-api-access-dlx8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:52.534183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.534164    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca" pod "a89b58d0-fa9b-41cc-8653-cee8298999e0" (UID: "a89b58d0-fa9b-41cc-8653-cee8298999e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:35:52.633461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.633425    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlx8r\" (UniqueName: \"kubernetes.io/projected/a89b58d0-fa9b-41cc-8653-cee8298999e0-kube-api-access-dlx8r\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:52.633461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:52.633456    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a89b58d0-fa9b-41cc-8653-cee8298999e0-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:35:53.276371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:53.276337    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"
Nov 02 18:35:53.276371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:53.276359    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn" event={"ID":"a89b58d0-fa9b-41cc-8653-cee8298999e0","Type":"ContainerDied","Data":"e068d6a724457244e41f1ee95d9b994b4438c87f7fb87de8b7c5d7dc65c6836d"}
Nov 02 18:35:53.276787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:53.276420    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e068d6a724457244e41f1ee95d9b994b4438c87f7fb87de8b7c5d7dc65c6836d"
Nov 02 18:35:54.646783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:54.646718    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:59.061612 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:35:59.061548    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:35:59.561293 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:35:59.561256    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:04.263460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:04.263427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:04.791417 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:04.791385    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:08.967872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:08.967835    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:13.605241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:13.605203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:17.681839 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:17.681800    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:19.355248 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:19.355207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:19.541972 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:19.541938    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:19.784655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:19.784588    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"]
Nov 02 18:36:19.784855 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:19.784829    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" containerID="cri-o://22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3" gracePeriod=30
Nov 02 18:36:19.825151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:19.825122    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:20.557587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:20.557551    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:22.695908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:22.695872    4476 patch_prober.go:29] interesting pod/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:36:22.696347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:22.695938    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:36:24.441980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:24.441939    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:25.067801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.067772    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:36:25.094813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.094786    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.094998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.094832    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.094998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.094875    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.094998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.094907    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.094998 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.094968    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.095222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095016    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.095222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095060    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:36:25.095222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095087    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.095222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095126    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.095422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095281    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token\") pod \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\" (UID: \"7d55225d-9bad-48ad-a7d8-3f0a741ec985\") "
Nov 02 18:36:25.095422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095294    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:36:25.095422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095308    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:36:25.095612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095590    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:36:25.095872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095850    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.096030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095898    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.096030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095919    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.096030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095933    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.096030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.095854    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:25.097870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.097845    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:36:25.098056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.098037    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "kube-api-access-gwzxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:25.098122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.098051    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:36:25.098122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.098060    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token" pod "7d55225d-9bad-48ad-a7d8-3f0a741ec985" (UID: "7d55225d-9bad-48ad-a7d8-3f0a741ec985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:25.196708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.196675    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.196708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.196707    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.196708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.196718    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.197015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.196727    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d55225d-9bad-48ad-a7d8-3f0a741ec985-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.197015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.196737    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwzxm\" (UniqueName: \"kubernetes.io/projected/7d55225d-9bad-48ad-a7d8-3f0a741ec985-kube-api-access-gwzxm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:25.544313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.544276    4476 generic.go:356] "Generic (PLEG): container finished" podID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerID="22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3" exitCode=0
Nov 02 18:36:25.544797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.544334    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"
Nov 02 18:36:25.544797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.544359    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" event={"ID":"7d55225d-9bad-48ad-a7d8-3f0a741ec985","Type":"ContainerDied","Data":"22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3"}
Nov 02 18:36:25.544797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.544425    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l" event={"ID":"7d55225d-9bad-48ad-a7d8-3f0a741ec985","Type":"ContainerDied","Data":"89acce6f8c10649bedcef28c4d9f4d4673cb7705883e9abe62c33388589ccbb0"}
Nov 02 18:36:25.544797 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.544450    4476 scope.go:122] "RemoveContainer" containerID="22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3"
Nov 02 18:36:25.556451 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.556428    4476 scope.go:122] "RemoveContainer" containerID="426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"
Nov 02 18:36:25.566484 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.566462    4476 scope.go:122] "RemoveContainer" containerID="22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3"
Nov 02 18:36:25.566859 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:25.566834    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3\": container with ID starting with 22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3 not found: ID does not exist" containerID="22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3"
Nov 02 18:36:25.566966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.566870    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3"} err="failed to get container status \"22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3\": rpc error: code = NotFound desc = could not find container \"22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3\": container with ID starting with 22139300be14c283408070b48856f6e57175881e0388fb3ed7a9d223886525b3 not found: ID does not exist"
Nov 02 18:36:25.566966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.566888    4476 scope.go:122] "RemoveContainer" containerID="426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"
Nov 02 18:36:25.567187 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:25.567158    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a\": container with ID starting with 426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a not found: ID does not exist" containerID="426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"
Nov 02 18:36:25.567247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.567194    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a"} err="failed to get container status \"426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a\": rpc error: code = NotFound desc = could not find container \"426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a\": container with ID starting with 426b75acdf6e0f7790420681d4c0a3ec646f3c15619562230f07b9c71fe7d64a not found: ID does not exist"
Nov 02 18:36:25.708010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.707905    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"]
Nov 02 18:36:25.758067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:25.758035    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-662a80-openshift-default-745684b69c-pqx9l"]
Nov 02 18:36:26.557380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:26.557346    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:26.745899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:26.745859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:26.938602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:26.938572    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:27.386041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:27.386007    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" path="/var/lib/kubelet/pods/7d55225d-9bad-48ad-a7d8-3f0a741ec985/volumes"
Nov 02 18:36:28.018714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.018681    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"]
Nov 02 18:36:28.332716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.332596    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:28.517102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.517049    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:28.597042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.596960    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"]
Nov 02 18:36:28.597213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.597180    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" podUID="ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" containerName="maas-api" containerID="cri-o://4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82" gracePeriod=30
Nov 02 18:36:28.763305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.763266    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:28.915707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:28.915360    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:36:29.057132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.041155    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5\" (UniqueName: \"kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5\") pod \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") "
Nov 02 18:36:29.057132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.041224    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca\") pod \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") "
Nov 02 18:36:29.057132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.041281    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs\") pod \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") "
Nov 02 18:36:29.057132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.041479    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\") pod \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\" (UID: \"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827\") "
Nov 02 18:36:29.057132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.046252    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca" pod "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" (UID: "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:29.077963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.077906    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5" pod "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" (UID: "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827"). InnerVolumeSpecName "kube-api-access-2bnt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:29.084751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.084086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs" pod "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" (UID: "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:36:29.085700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.085671    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls" pod "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" (UID: "ce22faf9-02ec-4aa3-a84c-7ad8c9f93827"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:36:29.104973 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:29.104940    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:36:29.145782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.145747    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:29.145782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.145783    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnt5\" (UniqueName: \"kubernetes.io/projected/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-kube-api-access-2bnt5\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:29.146006 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.145799    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:29.146006 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.145813    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:29.581745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.581706    4476 generic.go:356] "Generic (PLEG): container finished" podID="ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" containerID="4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82" exitCode=0
Nov 02 18:36:29.581745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.581751    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"
Nov 02 18:36:29.582028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.581791    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" event={"ID":"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827","Type":"ContainerDied","Data":"4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82"}
Nov 02 18:36:29.582028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.581849    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd" event={"ID":"ce22faf9-02ec-4aa3-a84c-7ad8c9f93827","Type":"ContainerDied","Data":"2978b9f50aae1e2221fe7c5fea066fbc22997bdc9563a64afea0b15f1e4d4517"}
Nov 02 18:36:29.582028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.581876    4476 scope.go:122] "RemoveContainer" containerID="4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82"
Nov 02 18:36:29.594771 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.594747    4476 scope.go:122] "RemoveContainer" containerID="4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82"
Nov 02 18:36:29.595145 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:29.595119    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82\": container with ID starting with 4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82 not found: ID does not exist" containerID="4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82"
Nov 02 18:36:29.595257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.595153    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82"} err="failed to get container status \"4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82\": rpc error: code = NotFound desc = could not find container \"4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82\": container with ID starting with 4c53368f822a3ea90fdb49aef64015a75bbaa491ad775fc0b2f4cb422fb40c82 not found: ID does not exist"
Nov 02 18:36:29.747382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.747277    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"]
Nov 02 18:36:29.753617 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:29.753583    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cf852-7b9b8555f6-f24cd"]
Nov 02 18:36:30.637264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:30.637230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:31.384737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:31.384704    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" path="/var/lib/kubelet/pods/ce22faf9-02ec-4aa3-a84c-7ad8c9f93827/volumes"
Nov 02 18:36:32.237596 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:32.237556    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:33.019478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.019426    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg" podUID="38735db1-7d04-4711-9dab-a0f55472982e" containerName="payload-pre-processing" containerID="cri-o://a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a" gracePeriod=25
Nov 02 18:36:33.285007 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.284940    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:36:33.397785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.397754    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume\") pod \"38735db1-7d04-4711-9dab-a0f55472982e\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") "
Nov 02 18:36:33.397970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.397866    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd\" (UniqueName: \"kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd\") pod \"38735db1-7d04-4711-9dab-a0f55472982e\" (UID: \"38735db1-7d04-4711-9dab-a0f55472982e\") "
Nov 02 18:36:33.398246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.398217    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume" pod "38735db1-7d04-4711-9dab-a0f55472982e" (UID: "38735db1-7d04-4711-9dab-a0f55472982e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:33.398619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.398595    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38735db1-7d04-4711-9dab-a0f55472982e-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:33.400677 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.400652    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd" pod "38735db1-7d04-4711-9dab-a0f55472982e" (UID: "38735db1-7d04-4711-9dab-a0f55472982e"). InnerVolumeSpecName "kube-api-access-p8kwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:33.499723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.499689    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kwd\" (UniqueName: \"kubernetes.io/projected/38735db1-7d04-4711-9dab-a0f55472982e-kube-api-access-p8kwd\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:33.614717 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.614683    4476 generic.go:356] "Generic (PLEG): container finished" podID="38735db1-7d04-4711-9dab-a0f55472982e" containerID="a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a" exitCode=0
Nov 02 18:36:33.614900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.614731    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"
Nov 02 18:36:33.614900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.614760    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg" event={"ID":"38735db1-7d04-4711-9dab-a0f55472982e","Type":"ContainerDied","Data":"a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a"}
Nov 02 18:36:33.614900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.614810    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg" event={"ID":"38735db1-7d04-4711-9dab-a0f55472982e","Type":"ContainerDied","Data":"c49c4c3fa4e46f1f900a6c494360102c567eeae4c4e6e0dcc47ba33ce366f3b2"}
Nov 02 18:36:33.614900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.614833    4476 scope.go:122] "RemoveContainer" containerID="a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a"
Nov 02 18:36:33.626086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.626067    4476 scope.go:122] "RemoveContainer" containerID="a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a"
Nov 02 18:36:33.626359 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:33.626340    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a\": container with ID starting with a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a not found: ID does not exist" containerID="a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a"
Nov 02 18:36:33.626405 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.626369    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a"} err="failed to get container status \"a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a\": rpc error: code = NotFound desc = could not find container \"a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a\": container with ID starting with a6dbc47af9816a15fb277fb357f3947f638c3ff7ac84cfe649dce9beaf88025a not found: ID does not exist"
Nov 02 18:36:33.779594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.779561    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"]
Nov 02 18:36:33.827335 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:33.827306    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cf852-7549b59595-zdhgg"]
Nov 02 18:36:35.301455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:35.301422    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:35.384022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:35.383988    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38735db1-7d04-4711-9dab-a0f55472982e" path="/var/lib/kubelet/pods/38735db1-7d04-4711-9dab-a0f55472982e/volumes"
Nov 02 18:36:38.138150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:38.138089    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:38.329048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:38.329004    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:39.613924 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:39.613895    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:39.802201 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:39.802159    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:40.079363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:40.079243    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:41.126143 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.126112    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:41.326193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.326162    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:41.483763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.483676    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"]
Nov 02 18:36:41.484336 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484316    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484338    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484365    4476 cpu_manager.go:414] "containerMap: removing container" podUID="a89b58d0-fa9b-41cc-8653-cee8298999e0" containerName="revoke-keys"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484373    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89b58d0-fa9b-41cc-8653-cee8298999e0" containerName="revoke-keys"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484391    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" containerName="maas-api"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484399    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" containerName="maas-api"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484422    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484429    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy"
Nov 02 18:36:41.484448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484446    4476 cpu_manager.go:414] "containerMap: removing container" podUID="38735db1-7d04-4711-9dab-a0f55472982e" containerName="payload-pre-processing"
Nov 02 18:36:41.484888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484454    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38735db1-7d04-4711-9dab-a0f55472982e" containerName="payload-pre-processing"
Nov 02 18:36:41.484888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484598    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89b58d0-fa9b-41cc-8653-cee8298999e0" containerName="revoke-keys"
Nov 02 18:36:41.484888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484614    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce22faf9-02ec-4aa3-a84c-7ad8c9f93827" containerName="maas-api"
Nov 02 18:36:41.484888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484645    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy"
Nov 02 18:36:41.484888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484655    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="38735db1-7d04-4711-9dab-a0f55472982e" containerName="payload-pre-processing"
Nov 02 18:36:41.484888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.484669    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d55225d-9bad-48ad-a7d8-3f0a741ec985" containerName="istio-proxy"
Nov 02 18:36:41.490043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.489997    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.547433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.547404    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"]
Nov 02 18:36:41.592685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.592658    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:36:41.592851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.592711    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:36:41.599272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.599248    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.599412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.599394    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q6w\" (UniqueName: \"kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.599462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.599449    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.659552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.659518    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:41.700650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.700599    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.700861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.700783    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q6w\" (UniqueName: \"kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.700948 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.700870    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.701770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.701711    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.703737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.703715    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.761141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.761048    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q6w\" (UniqueName: \"kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w\") pod \"maas-api-revoke-sub-77fa407c-v6gfm\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.803698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.803664    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:41.837964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:41.837932    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:42.157029 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:42.156993    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:42.188430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:42.188396    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"]
Nov 02 18:36:42.189306 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:36:42.189270    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf948e1e6_7557_414e_aabd_f44b92178700.slice/crio-a8d543d84f5a0c762a66882e7e78a63eb442e91c185d6d73f11a3cd7c0c520ea WatchSource:0}: Error finding container a8d543d84f5a0c762a66882e7e78a63eb442e91c185d6d73f11a3cd7c0c520ea: Status 404 returned error can't find the container with id a8d543d84f5a0c762a66882e7e78a63eb442e91c185d6d73f11a3cd7c0c520ea
Nov 02 18:36:42.688701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:42.688663    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm" event={"ID":"f948e1e6-7557-414e-aabd-f44b92178700","Type":"ContainerStarted","Data":"e2c42b490aba9b2afba0af3f5d83b7113d0cc4dc353cc528d716bafa5c8cc9cf"}
Nov 02 18:36:42.688867 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:42.688727    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm" event={"ID":"f948e1e6-7557-414e-aabd-f44b92178700","Type":"ContainerStarted","Data":"a8d543d84f5a0c762a66882e7e78a63eb442e91c185d6d73f11a3cd7c0c520ea"}
Nov 02 18:36:42.808355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:42.808255    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm" podStartSLOduration=1.808240386 podStartE2EDuration="1.808240386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:42.805691967 +0000 UTC m=+4433.646297901" watchObservedRunningTime="2026-11-02 18:36:42.808240386 +0000 UTC m=+4433.648846314"
Nov 02 18:36:42.903328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:42.903289    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:43.697232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:43.697194    4476 generic.go:356] "Generic (PLEG): container finished" podID="f948e1e6-7557-414e-aabd-f44b92178700" containerID="e2c42b490aba9b2afba0af3f5d83b7113d0cc4dc353cc528d716bafa5c8cc9cf" exitCode=0
Nov 02 18:36:43.697655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:43.697272    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm" event={"ID":"f948e1e6-7557-414e-aabd-f44b92178700","Type":"ContainerDied","Data":"e2c42b490aba9b2afba0af3f5d83b7113d0cc4dc353cc528d716bafa5c8cc9cf"}
Nov 02 18:36:44.585525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.585475    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:44.797953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.797915    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:44.858050 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.858028    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:44.954170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.954142    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca\") pod \"f948e1e6-7557-414e-aabd-f44b92178700\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") "
Nov 02 18:36:44.954328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.954178    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w\" (UniqueName: \"kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w\") pod \"f948e1e6-7557-414e-aabd-f44b92178700\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") "
Nov 02 18:36:44.954328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.954245    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca\") pod \"f948e1e6-7557-414e-aabd-f44b92178700\" (UID: \"f948e1e6-7557-414e-aabd-f44b92178700\") "
Nov 02 18:36:44.954532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.954510    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca" pod "f948e1e6-7557-414e-aabd-f44b92178700" (UID: "f948e1e6-7557-414e-aabd-f44b92178700"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:44.954767 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.954746    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:44.956788 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.956757    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w" pod "f948e1e6-7557-414e-aabd-f44b92178700" (UID: "f948e1e6-7557-414e-aabd-f44b92178700"). InnerVolumeSpecName "kube-api-access-74q6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:44.956914 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:44.956856    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca" pod "f948e1e6-7557-414e-aabd-f44b92178700" (UID: "f948e1e6-7557-414e-aabd-f44b92178700"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:36:45.055825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:45.055789    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74q6w\" (UniqueName: \"kubernetes.io/projected/f948e1e6-7557-414e-aabd-f44b92178700-kube-api-access-74q6w\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:45.055825 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:45.055818    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f948e1e6-7557-414e-aabd-f44b92178700-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:45.713446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:45.713372    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"
Nov 02 18:36:45.713446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:45.713413    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm" event={"ID":"f948e1e6-7557-414e-aabd-f44b92178700","Type":"ContainerDied","Data":"a8d543d84f5a0c762a66882e7e78a63eb442e91c185d6d73f11a3cd7c0c520ea"}
Nov 02 18:36:45.713662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:45.713458    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8d543d84f5a0c762a66882e7e78a63eb442e91c185d6d73f11a3cd7c0c520ea"
Nov 02 18:36:46.941803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:46.941773    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:48.677644 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:48.677600    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:49.613871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:49.613835    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"]
Nov 02 18:36:49.720766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:49.720733    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"]
Nov 02 18:36:50.021291 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:50.021211    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:50.200826 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:50.200793    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:51.457423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:51.457387    4476 scope.go:122] "RemoveContainer" containerID="7dbb02e1042d3a8db247027e32c63b2749351619de4b1a814cf649661b859705"
Nov 02 18:36:51.535067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:51.535041    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:52.394363 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:52.394334    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:53.690526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:53.690490    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:54.450205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.450171    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:54.614992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.614948    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9" podUID="4aec8aaf-241f-43bb-8e6b-afa034a9da2c" containerName="payload-processing" containerID="cri-o://f840d233e8ef4c593fbf8fc6ce78eb55bd7724ce45b5f1c4c96861795a64c343" gracePeriod=25
Nov 02 18:36:54.721974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.721893    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj" podUID="ca107b84-5b8c-4781-bab0-49215ab7ccbf" containerName="payload-pre-processing" containerID="cri-o://15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323" gracePeriod=25
Nov 02 18:36:54.795889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.795860    4476 generic.go:356] "Generic (PLEG): container finished" podID="4aec8aaf-241f-43bb-8e6b-afa034a9da2c" containerID="f840d233e8ef4c593fbf8fc6ce78eb55bd7724ce45b5f1c4c96861795a64c343" exitCode=0
Nov 02 18:36:54.796079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.795963    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9" event={"ID":"4aec8aaf-241f-43bb-8e6b-afa034a9da2c","Type":"ContainerDied","Data":"f840d233e8ef4c593fbf8fc6ce78eb55bd7724ce45b5f1c4c96861795a64c343"}
Nov 02 18:36:54.893619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.893591    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:36:54.985193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.978007    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2\" (UniqueName: \"kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2\") pod \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") "
Nov 02 18:36:54.985193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.978269    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume\") pod \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\" (UID: \"4aec8aaf-241f-43bb-8e6b-afa034a9da2c\") "
Nov 02 18:36:54.991651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:54.988856    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume" pod "4aec8aaf-241f-43bb-8e6b-afa034a9da2c" (UID: "4aec8aaf-241f-43bb-8e6b-afa034a9da2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:55.011932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.009411    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2" pod "4aec8aaf-241f-43bb-8e6b-afa034a9da2c" (UID: "4aec8aaf-241f-43bb-8e6b-afa034a9da2c"). InnerVolumeSpecName "kube-api-access-nr8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:55.038678 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.038651    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:36:55.079598 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.079562    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume\") pod \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") "
Nov 02 18:36:55.079799 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.079666    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc\" (UniqueName: \"kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc\") pod \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\" (UID: \"ca107b84-5b8c-4781-bab0-49215ab7ccbf\") "
Nov 02 18:36:55.079918 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.079897    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume" pod "ca107b84-5b8c-4781-bab0-49215ab7ccbf" (UID: "ca107b84-5b8c-4781-bab0-49215ab7ccbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:55.080138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.080124    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8k2\" (UniqueName: \"kubernetes.io/projected/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-kube-api-access-nr8k2\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:55.080184 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.080141    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca107b84-5b8c-4781-bab0-49215ab7ccbf-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:55.080184 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.080150    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aec8aaf-241f-43bb-8e6b-afa034a9da2c-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:55.081864 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.081842    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc" pod "ca107b84-5b8c-4781-bab0-49215ab7ccbf" (UID: "ca107b84-5b8c-4781-bab0-49215ab7ccbf"). InnerVolumeSpecName "kube-api-access-7hxnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:55.181272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.181239    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxnc\" (UniqueName: \"kubernetes.io/projected/ca107b84-5b8c-4781-bab0-49215ab7ccbf-kube-api-access-7hxnc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:36:55.804870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.804766    4476 generic.go:356] "Generic (PLEG): container finished" podID="ca107b84-5b8c-4781-bab0-49215ab7ccbf" containerID="15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323" exitCode=0
Nov 02 18:36:55.805344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.804881    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"
Nov 02 18:36:55.805344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.804908    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj" event={"ID":"ca107b84-5b8c-4781-bab0-49215ab7ccbf","Type":"ContainerDied","Data":"15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323"}
Nov 02 18:36:55.805344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.804977    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj" event={"ID":"ca107b84-5b8c-4781-bab0-49215ab7ccbf","Type":"ContainerDied","Data":"82f31e68e1305113e13dc3ab9698393070a1cb33359b68ce34969aba16e23520"}
Nov 02 18:36:55.805344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.805011    4476 scope.go:122] "RemoveContainer" containerID="15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323"
Nov 02 18:36:55.806682 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.806664    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"
Nov 02 18:36:55.806865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.806657    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9" event={"ID":"4aec8aaf-241f-43bb-8e6b-afa034a9da2c","Type":"ContainerDied","Data":"a8f46abe3439d84411c46b27915a0b0c59823d03e700f420ed8c51917988651f"}
Nov 02 18:36:55.816251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.816233    4476 scope.go:122] "RemoveContainer" containerID="15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323"
Nov 02 18:36:55.816533 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:55.816507    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323\": container with ID starting with 15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323 not found: ID does not exist" containerID="15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323"
Nov 02 18:36:55.816590 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.816530    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323"} err="failed to get container status \"15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323\": rpc error: code = NotFound desc = could not find container \"15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323\": container with ID starting with 15c02a9974776922265d14f3f7025a7fcc13c1fd1a1255b1bf56c599fc4ff323 not found: ID does not exist"
Nov 02 18:36:55.816590 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.816547    4476 scope.go:122] "RemoveContainer" containerID="f840d233e8ef4c593fbf8fc6ce78eb55bd7724ce45b5f1c4c96861795a64c343"
Nov 02 18:36:55.968144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:55.968114    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"]
Nov 02 18:36:56.017380 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:56.017352    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a51e27-bdcf6c89c-pxsr9"]
Nov 02 18:36:56.082054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:56.081915    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"]
Nov 02 18:36:56.085353 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:56.085328    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a51e27-fcd7f6495-lnbpj"]
Nov 02 18:36:57.384327 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:57.384289    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aec8aaf-241f-43bb-8e6b-afa034a9da2c" path="/var/lib/kubelet/pods/4aec8aaf-241f-43bb-8e6b-afa034a9da2c/volumes"
Nov 02 18:36:57.384911 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:57.384890    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca107b84-5b8c-4781-bab0-49215ab7ccbf" path="/var/lib/kubelet/pods/ca107b84-5b8c-4781-bab0-49215ab7ccbf/volumes"
Nov 02 18:36:57.440561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:57.440523    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:58.107983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:58.107952    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:58.291541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:36:58.291508    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:36:59.060323 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:36:59.060286    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:37:02.782974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:02.782877    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:03.380031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:03.379995    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:05.132922 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:05.132885    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:07.650134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:07.650098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:08.836763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:08.836722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:09.241622 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:09.241591    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:09.830464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:09.830418    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:10.149284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:10.149251    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:12.640919 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:12.640889    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:14.476078 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:14.476039    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:14.661337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:14.661276    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:16.869575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:16.869537    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"]
Nov 02 18:37:16.870120 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:16.869845    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" podUID="585ed752-b048-4c68-9713-2892dad85ad5" containerName="maas-api" containerID="cri-o://84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf" gracePeriod=30
Nov 02 18:37:17.015252 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.015215    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:17.172661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.172619    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:37:17.350602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.350568    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\") pod \"585ed752-b048-4c68-9713-2892dad85ad5\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") "
Nov 02 18:37:17.350993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.350971    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn\" (UniqueName: \"kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn\") pod \"585ed752-b048-4c68-9713-2892dad85ad5\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") "
Nov 02 18:37:17.351161 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.351145    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs\") pod \"585ed752-b048-4c68-9713-2892dad85ad5\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") "
Nov 02 18:37:17.351296 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.351280    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca\") pod \"585ed752-b048-4c68-9713-2892dad85ad5\" (UID: \"585ed752-b048-4c68-9713-2892dad85ad5\") "
Nov 02 18:37:17.352235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.352213    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca" pod "585ed752-b048-4c68-9713-2892dad85ad5" (UID: "585ed752-b048-4c68-9713-2892dad85ad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:17.369553 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.369505    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls" pod "585ed752-b048-4c68-9713-2892dad85ad5" (UID: "585ed752-b048-4c68-9713-2892dad85ad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:37:17.371881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.371858    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn" pod "585ed752-b048-4c68-9713-2892dad85ad5" (UID: "585ed752-b048-4c68-9713-2892dad85ad5"). InnerVolumeSpecName "kube-api-access-6vxkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:17.372800 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.372760    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs" pod "585ed752-b048-4c68-9713-2892dad85ad5" (UID: "585ed752-b048-4c68-9713-2892dad85ad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:37:17.452871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.452782    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:17.452871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.452821    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxkn\" (UniqueName: \"kubernetes.io/projected/585ed752-b048-4c68-9713-2892dad85ad5-kube-api-access-6vxkn\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:17.452871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.452837    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585ed752-b048-4c68-9713-2892dad85ad5-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:17.452871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.452852    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585ed752-b048-4c68-9713-2892dad85ad5-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:17.743032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.742944    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:17.935104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.935075    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:17.974258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.974219    4476 generic.go:356] "Generic (PLEG): container finished" podID="585ed752-b048-4c68-9713-2892dad85ad5" containerID="84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf" exitCode=0
Nov 02 18:37:17.974466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.974326    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"
Nov 02 18:37:17.974466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.974316    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" event={"ID":"585ed752-b048-4c68-9713-2892dad85ad5","Type":"ContainerDied","Data":"84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf"}
Nov 02 18:37:17.974466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.974436    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb" event={"ID":"585ed752-b048-4c68-9713-2892dad85ad5","Type":"ContainerDied","Data":"16ccda3f6b24f36044b9e2684db9020989d1065214d322493678e6f135146b82"}
Nov 02 18:37:17.974466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.974467    4476 scope.go:122] "RemoveContainer" containerID="84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf"
Nov 02 18:37:17.992227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.992194    4476 scope.go:122] "RemoveContainer" containerID="84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf"
Nov 02 18:37:17.992592 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:37:17.992565    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf\": container with ID starting with 84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf not found: ID does not exist" containerID="84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf"
Nov 02 18:37:17.992724 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:17.992601    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf"} err="failed to get container status \"84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf\": rpc error: code = NotFound desc = could not find container \"84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf\": container with ID starting with 84e8eb8f63a1a06151f8c287f6922097c4485a0c8e254d360b075db9a3809aaf not found: ID does not exist"
Nov 02 18:37:18.105240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105153    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"]
Nov 02 18:37:18.105737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105722    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ca107b84-5b8c-4781-bab0-49215ab7ccbf" containerName="payload-pre-processing"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105738    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca107b84-5b8c-4781-bab0-49215ab7ccbf" containerName="payload-pre-processing"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105752    4476 cpu_manager.go:414] "containerMap: removing container" podUID="585ed752-b048-4c68-9713-2892dad85ad5" containerName="maas-api"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105757    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585ed752-b048-4c68-9713-2892dad85ad5" containerName="maas-api"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105779    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f948e1e6-7557-414e-aabd-f44b92178700" containerName="revoke-keys"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105785    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f948e1e6-7557-414e-aabd-f44b92178700" containerName="revoke-keys"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105800    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4aec8aaf-241f-43bb-8e6b-afa034a9da2c" containerName="payload-processing"
Nov 02 18:37:18.105824 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105808    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aec8aaf-241f-43bb-8e6b-afa034a9da2c" containerName="payload-processing"
Nov 02 18:37:18.106191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105908    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca107b84-5b8c-4781-bab0-49215ab7ccbf" containerName="payload-pre-processing"
Nov 02 18:37:18.106191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105917    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="585ed752-b048-4c68-9713-2892dad85ad5" containerName="maas-api"
Nov 02 18:37:18.106191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105929    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f948e1e6-7557-414e-aabd-f44b92178700" containerName="revoke-keys"
Nov 02 18:37:18.106191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.105935    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aec8aaf-241f-43bb-8e6b-afa034a9da2c" containerName="payload-processing"
Nov 02 18:37:18.110665 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.110644    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.141478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.141441    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"]
Nov 02 18:37:18.145762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.145736    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-740c43-6cd89dd9db-hwrrb"]
Nov 02 18:37:18.161271 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.161226    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qlq\" (UniqueName: \"kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq\") pod \"payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.161424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.161376    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.162605 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.162587    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4fe24d\""
Nov 02 18:37:18.162719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.162711    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4fe24d-dockercfg-mm6vp\""
Nov 02 18:37:18.167382 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.167358    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"]
Nov 02 18:37:18.277216 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.277170    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qlq\" (UniqueName: \"kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq\") pod \"payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.277427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.277254    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.278372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.278342    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.280930 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.280912    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"]
Nov 02 18:37:18.305696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.305664    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.317163 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.317135    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"]
Nov 02 18:37:18.334432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.334406    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qlq\" (UniqueName: \"kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq\") pod \"payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.378508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.378461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.378697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.378554    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvj8k\" (UniqueName: \"kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k\") pod \"payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.424255 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.424223    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:18.482251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.482213    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvj8k\" (UniqueName: \"kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k\") pod \"payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.482462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.482441    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.483403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.483379    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.540311 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.540274    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvj8k\" (UniqueName: \"kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k\") pod \"payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.622013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.621990    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:18.811059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.811030    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"]
Nov 02 18:37:18.811801 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:37:18.811772    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80b6429_fd80_4c59_a1a6_b0b991667f03.slice/crio-c8abcb4d518eef3271bf9f80352285a6d2fed8711cbfc3e5b08a5f362abf7804 WatchSource:0}: Error finding container c8abcb4d518eef3271bf9f80352285a6d2fed8711cbfc3e5b08a5f362abf7804: Status 404 returned error can't find the container with id c8abcb4d518eef3271bf9f80352285a6d2fed8711cbfc3e5b08a5f362abf7804
Nov 02 18:37:18.858226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.858176    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:18.985909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.985873    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" event={"ID":"b80b6429-fd80-4c59-a1a6-b0b991667f03","Type":"ContainerStarted","Data":"0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48"}
Nov 02 18:37:18.986426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.985971    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" event={"ID":"b80b6429-fd80-4c59-a1a6-b0b991667f03","Type":"ContainerStarted","Data":"c8abcb4d518eef3271bf9f80352285a6d2fed8711cbfc3e5b08a5f362abf7804"}
Nov 02 18:37:18.986426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:18.986004    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:19.011908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.011883    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"]
Nov 02 18:37:19.013532 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:37:19.013508    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b823f43_2b94_4f45_a604_fed01945ae56.slice/crio-c9ea747e7bd13e3e2e1c290495195783fad38f23b5b2870f5a3e82f808636a62 WatchSource:0}: Error finding container c9ea747e7bd13e3e2e1c290495195783fad38f23b5b2870f5a3e82f808636a62: Status 404 returned error can't find the container with id c9ea747e7bd13e3e2e1c290495195783fad38f23b5b2870f5a3e82f808636a62
Nov 02 18:37:19.061651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.061263    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:19.067790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.067766    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"]
Nov 02 18:37:19.072545 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.072523    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-740c43-gpnsn"]
Nov 02 18:37:19.100129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.100082    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" podStartSLOduration=1.100068713 podStartE2EDuration="1.100068713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:37:19.098597076 +0000 UTC m=+4469.939203011" watchObservedRunningTime="2026-11-02 18:37:19.100068713 +0000 UTC m=+4469.940674647"
Nov 02 18:37:19.277312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.277212    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"]
Nov 02 18:37:19.277501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.277460    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy" containerID="cri-o://8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0" gracePeriod=30
Nov 02 18:37:19.366433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.366402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:19.387157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.387118    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585ed752-b048-4c68-9713-2892dad85ad5" path="/var/lib/kubelet/pods/585ed752-b048-4c68-9713-2892dad85ad5/volumes"
Nov 02 18:37:19.387862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.387840    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89b58d0-fa9b-41cc-8653-cee8298999e0" path="/var/lib/kubelet/pods/a89b58d0-fa9b-41cc-8653-cee8298999e0/volumes"
Nov 02 18:37:19.995837 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.995798    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8" event={"ID":"4b823f43-2b94-4f45-a604-fed01945ae56","Type":"ContainerStarted","Data":"80dd1d0bf2bc92c57336b8aa7057f2395e0f6e6cc7731fe56715bd89b40f051d"}
Nov 02 18:37:19.996357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.995856    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8" event={"ID":"4b823f43-2b94-4f45-a604-fed01945ae56","Type":"ContainerStarted","Data":"c9ea747e7bd13e3e2e1c290495195783fad38f23b5b2870f5a3e82f808636a62"}
Nov 02 18:37:19.996357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:19.996027    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:20.095194 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:20.095162    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:20.111885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:20.111832    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8" podStartSLOduration=2.111818699 podStartE2EDuration="2.111818699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:37:20.108140311 +0000 UTC m=+4470.948746244" watchObservedRunningTime="2026-11-02 18:37:20.111818699 +0000 UTC m=+4470.952424633"
Nov 02 18:37:21.748641 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:21.748592    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:22.001178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:22.001098    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:24.085786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.085741    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:24.594242 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.594204    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667082    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667227    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667261    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667295    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667340    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667385    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667426    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667498    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.667551    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo\") pod \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\" (UID: \"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9\") "
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.668686    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.668877    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.669143    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.669553    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.669736    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:37:24.671703 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.671045    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:37:24.672834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.672779    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:37:24.672834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.672800    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:24.673104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.673080    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc" pod "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" (UID: "16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9"). InnerVolumeSpecName "kube-api-access-mkdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:24.769091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769059    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkdlc\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-kube-api-access-mkdlc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769089    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769103    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769116    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769127    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769138    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769149    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769161    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:24.769323 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:24.769173    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:37:25.030032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.029943    4476 generic.go:356] "Generic (PLEG): container finished" podID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerID="8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0" exitCode=0
Nov 02 18:37:25.030032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.030003    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"
Nov 02 18:37:25.030285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.030029    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" event={"ID":"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9","Type":"ContainerDied","Data":"8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0"}
Nov 02 18:37:25.030285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.030088    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb" event={"ID":"16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9","Type":"ContainerDied","Data":"2e1182459f9a13032f7aee4450471fbf73fcba292fd8dc0a11751e4309688bf2"}
Nov 02 18:37:25.030285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.030118    4476 scope.go:122] "RemoveContainer" containerID="8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0"
Nov 02 18:37:25.041800 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.041778    4476 scope.go:122] "RemoveContainer" containerID="273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"
Nov 02 18:37:25.051284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.051265    4476 scope.go:122] "RemoveContainer" containerID="8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0"
Nov 02 18:37:25.051568 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:37:25.051548    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0\": container with ID starting with 8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0 not found: ID does not exist" containerID="8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0"
Nov 02 18:37:25.051671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.051577    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0"} err="failed to get container status \"8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0\": rpc error: code = NotFound desc = could not find container \"8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0\": container with ID starting with 8848fe753eb12751438eec83f77f563ca052c8657d5cbd9fdbc138a8a650bfa0 not found: ID does not exist"
Nov 02 18:37:25.051671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.051598    4476 scope.go:122] "RemoveContainer" containerID="273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"
Nov 02 18:37:25.051863 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:37:25.051846    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26\": container with ID starting with 273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26 not found: ID does not exist" containerID="273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"
Nov 02 18:37:25.051902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.051872    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26"} err="failed to get container status \"273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26\": rpc error: code = NotFound desc = could not find container \"273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26\": container with ID starting with 273cbe5224270712f3922a34764397c3c9adc55ca6fa52f4a429fba8a73aba26 not found: ID does not exist"
Nov 02 18:37:25.195671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.195618    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"]
Nov 02 18:37:25.243359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.243329    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-740c43-openshift-default-7d8b9c5d85-l76qb"]
Nov 02 18:37:25.384556 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.384525    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" path="/var/lib/kubelet/pods/16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9/volumes"
Nov 02 18:37:25.785852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:25.785769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:26.051852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:26.051759    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:26.713039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:26.712961    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:29.061175 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:37:29.061140    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:37:29.346432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:29.346338    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:29.999027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:29.998996    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:37:31.006139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:31.006111    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:37:32.132639 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:32.132589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:34.074530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:34.074481    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:38.712065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:38.712015    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:43.369828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:43.369769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:48.059973 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:48.059920    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:50.169700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:50.169672    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:37:50.178099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:50.178069    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:37:52.757192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:52.757133    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:57.449220 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:37:57.449168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:37:59.061387 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:37:59.061340    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:38:02.153480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:02.153431    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:05.475217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:05.475176    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:06.858358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:06.858312    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:07.568339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:07.568303    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:07.748964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:07.748929    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:12.740353 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:12.740319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:14.425217 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:14.425179    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:14.918531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:14.918498    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:15.874570 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:15.874527    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:16.080071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:16.080031    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:17.257999 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:17.257966    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:17.453227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:17.453186    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:17.963260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:17.963225    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:19.295352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.295320    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"]
Nov 02 18:38:19.295856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.295843    4476 cpu_manager.go:414] "containerMap: removing container" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy"
Nov 02 18:38:19.295903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.295857    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy"
Nov 02 18:38:19.295903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.295882    4476 cpu_manager.go:414] "containerMap: removing container" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy"
Nov 02 18:38:19.295903 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.295887    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy"
Nov 02 18:38:19.296018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.295985    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy"
Nov 02 18:38:19.299360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.299343    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.354719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.354688    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"]
Nov 02 18:38:19.387440 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.387405    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:38:19.399373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.399347    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:38:19.418767 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.418741    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjfg\" (UniqueName: \"kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.418945 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.418783    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.418945 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.418868    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.520062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.520027    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjfg\" (UniqueName: \"kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.520267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.520070    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.520267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.520189    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.520858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.520835    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.522956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.522936    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.576482 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.576410    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjfg\" (UniqueName: \"kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg\") pod \"maas-api-revoke-sub-4f3e8628-jtfsh\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.620577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.620548    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:19.836287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.836199    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:19.998203 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:19.998175    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"]
Nov 02 18:38:19.999256 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:38:19.999230    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699a874b_ec31_4413_af94_e29840916ed1.slice/crio-d4d836dcfea164500d858ee0a7429f45a47c49d1ccb40ed7f3a500f194604621 WatchSource:0}: Error finding container d4d836dcfea164500d858ee0a7429f45a47c49d1ccb40ed7f3a500f194604621: Status 404 returned error can't find the container with id d4d836dcfea164500d858ee0a7429f45a47c49d1ccb40ed7f3a500f194604621
Nov 02 18:38:20.029245 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:20.029216    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:20.446994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:20.446889    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh" event={"ID":"699a874b-ec31-4413-af94-e29840916ed1","Type":"ContainerStarted","Data":"9b6cec547dfdce53915f6c5cc1279037ba3605e3edd35cd80affba41cc29e35d"}
Nov 02 18:38:20.446994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:20.446952    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh" event={"ID":"699a874b-ec31-4413-af94-e29840916ed1","Type":"ContainerStarted","Data":"d4d836dcfea164500d858ee0a7429f45a47c49d1ccb40ed7f3a500f194604621"}
Nov 02 18:38:20.561714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:20.561658    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh" podStartSLOduration=1.561641082 podStartE2EDuration="1.561641082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:38:20.558278627 +0000 UTC m=+4531.398884562" watchObservedRunningTime="2026-11-02 18:38:20.561641082 +0000 UTC m=+4531.402247015"
Nov 02 18:38:21.455606 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:21.455567    4476 generic.go:356] "Generic (PLEG): container finished" podID="699a874b-ec31-4413-af94-e29840916ed1" containerID="9b6cec547dfdce53915f6c5cc1279037ba3605e3edd35cd80affba41cc29e35d" exitCode=0
Nov 02 18:38:21.456142 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:21.455659    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh" event={"ID":"699a874b-ec31-4413-af94-e29840916ed1","Type":"ContainerDied","Data":"9b6cec547dfdce53915f6c5cc1279037ba3605e3edd35cd80affba41cc29e35d"}
Nov 02 18:38:22.605817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.605789    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:22.652455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.652431    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg\" (UniqueName: \"kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg\") pod \"699a874b-ec31-4413-af94-e29840916ed1\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") "
Nov 02 18:38:22.652600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.652502    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca\") pod \"699a874b-ec31-4413-af94-e29840916ed1\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") "
Nov 02 18:38:22.652600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.652550    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca\") pod \"699a874b-ec31-4413-af94-e29840916ed1\" (UID: \"699a874b-ec31-4413-af94-e29840916ed1\") "
Nov 02 18:38:22.652876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.652849    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca" pod "699a874b-ec31-4413-af94-e29840916ed1" (UID: "699a874b-ec31-4413-af94-e29840916ed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:38:22.654749 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.654721    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca" pod "699a874b-ec31-4413-af94-e29840916ed1" (UID: "699a874b-ec31-4413-af94-e29840916ed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:38:22.654856 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.654773    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg" pod "699a874b-ec31-4413-af94-e29840916ed1" (UID: "699a874b-ec31-4413-af94-e29840916ed1"). InnerVolumeSpecName "kube-api-access-nrjfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:38:22.753920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.753824    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/699a874b-ec31-4413-af94-e29840916ed1-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:22.753920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.753855    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/699a874b-ec31-4413-af94-e29840916ed1-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:22.753920 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.753864    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjfg\" (UniqueName: \"kubernetes.io/projected/699a874b-ec31-4413-af94-e29840916ed1-kube-api-access-nrjfg\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:38:22.839154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:22.839122    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:23.032488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:23.032369    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:23.470297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:23.470261    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh" event={"ID":"699a874b-ec31-4413-af94-e29840916ed1","Type":"ContainerDied","Data":"d4d836dcfea164500d858ee0a7429f45a47c49d1ccb40ed7f3a500f194604621"}
Nov 02 18:38:23.470455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:23.470302    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"
Nov 02 18:38:23.470455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:23.470308    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d836dcfea164500d858ee0a7429f45a47c49d1ccb40ed7f3a500f194604621"
Nov 02 18:38:23.752676 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:23.752574    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:27.708447 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:27.708416    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:27.890100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:27.890066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:29.059993 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:38:29.059953    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:38:30.040205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:30.040167    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:31.858527 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.858451    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"]
Nov 02 18:38:31.858983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.858969    4476 cpu_manager.go:414] "containerMap: removing container" podUID="699a874b-ec31-4413-af94-e29840916ed1" containerName="revoke-keys"
Nov 02 18:38:31.859037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.858985    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699a874b-ec31-4413-af94-e29840916ed1" containerName="revoke-keys"
Nov 02 18:38:31.859107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.859098    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="699a874b-ec31-4413-af94-e29840916ed1" containerName="revoke-keys"
Nov 02 18:38:31.859144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.859111    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bb92d5-dd8a-4c83-bd9b-e8d1f54af9a9" containerName="istio-proxy"
Nov 02 18:38:31.862474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.862458    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:31.955157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.955128    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:31.955341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.955193    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:31.955341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.955270    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52q88\" (UniqueName: \"kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:31.955463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.955437    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:31.963674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.963650    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4fe24d-serving-cert\""
Nov 02 18:38:31.967401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:31.967379    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"]
Nov 02 18:38:32.056935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.056898    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.057110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.056979    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.057110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.056996    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52q88\" (UniqueName: \"kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.057110 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:38:32.057001    4476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-serving-cert: secret "maas-api-e2e-inf-b-4fe24d-serving-cert" not found
Nov 02 18:38:32.057110 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.057045    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.057110 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:38:32.057059    4476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls podName:0b26c2bf-2217-4f94-9005-b739811a8667 nodeName:}" failed. No retries permitted until 2026-11-02 18:38:32.557043637 +0000 UTC m=+4543.397649549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls") pod "maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" (UID: "0b26c2bf-2217-4f94-9005-b739811a8667") : secret "maas-api-e2e-inf-b-4fe24d-serving-cert" not found
Nov 02 18:38:32.057667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.057648    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.059564 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.059530    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.116947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.116915    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52q88\" (UniqueName: \"kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.132763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.132693    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:32.315157 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.315102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:32.565068 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.564966    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.577537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.577504    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\") pod \"maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:32.775845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:32.775807    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:33.173573 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:33.173538    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"]
Nov 02 18:38:33.179855 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:33.179828    4476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:38:33.600712 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:33.599795    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" event={"ID":"0b26c2bf-2217-4f94-9005-b739811a8667","Type":"ContainerStarted","Data":"e6e62866e4cc4ff52b0113b2c25835289653e36b58d5add361c1047317131e59"}
Nov 02 18:38:34.625004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:34.624961    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" event={"ID":"0b26c2bf-2217-4f94-9005-b739811a8667","Type":"ContainerStarted","Data":"c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7"}
Nov 02 18:38:34.625448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:34.625120    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:34.752653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:34.748871    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" podStartSLOduration=2.775013222 podStartE2EDuration="3.74885114s" totalImagesPullingTime="973.837918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:38:34.746030301 +0000 UTC m=+4545.586636236" watchObservedRunningTime="2026-11-02 18:38:34.74885114 +0000 UTC m=+4545.589457075"
Nov 02 18:38:35.679408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:35.679367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:37.121065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:37.121028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:38.455551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:38.455482    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:40.637792 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:40.637763    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:38:41.785801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:41.785709    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:44.469453 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:44.469419    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:45.835932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:45.835893    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:47.597866 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:47.597828    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:49.976491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:49.976455    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:50.348992 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:50.348899    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:51.625329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:51.625266    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:52.072215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:52.072123    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:52.440961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:52.440926    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:55.373318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:55.373283    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:55.568236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:55.568199    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:56.687531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:56.687493    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:57.411509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:38:57.411471    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:38:59.059438 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:38:59.059397    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:39:02.064204 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:02.064168    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:05.894894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:05.894863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:07.306655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:07.305674    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:39:07.306655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:07.305750    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:39:07.360654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:07.359784    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:39:07.360654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:07.359868    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:39:07.470427 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:07.470115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:12.279621 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:12.279582    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:17.028597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:17.028541    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:21.747325 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:21.747275    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:26.387685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:26.387620    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:29.060883 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:39:29.060840    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:39:31.075251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:31.075201    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:35.789539 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:35.789498    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:40.489388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:40.489273    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:45.150871 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:45.150830    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:49.856771 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:49.856721    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:54.564304 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:54.564267    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:57.895461 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:57.895429    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:39:59.061526 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:39:59.061485    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:39:59.492696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:39:59.492596    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:01.701964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.701880    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"]
Nov 02 18:40:01.705851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.705833    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.759878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.759844    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-134555\"/\"openshift-service-ca.crt\""
Nov 02 18:40:01.786415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.786382    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.786575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.786524    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.786647 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.786598    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.786708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.786645    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.786708 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.786664    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstk9\" (UniqueName: \"kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.786819 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.786800    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.808987 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.808964    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-134555\"/\"kube-root-ca.crt\""
Nov 02 18:40:01.809168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.809150    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-134555\"/\"default-dockercfg-h9lnb\""
Nov 02 18:40:01.809211 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.809151    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-134555\"/\"test-model-134555-kserve-self-signed-certs\""
Nov 02 18:40:01.812928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.812911    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"]
Nov 02 18:40:01.846460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.846406    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:01.888227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888189    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888272    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888371    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888416    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888448    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888473    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstk9\" (UniqueName: \"kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888781    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888843    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.888956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.888886    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.890798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.890773    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.891011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.890994    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:01.945881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:01.945843    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstk9\" (UniqueName: \"kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9\") pod \"test-model-134555-kserve-5b94869b9c-nspvn\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") " pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:02.018695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:02.018605    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:02.412788 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:02.412758    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"]
Nov 02 18:40:02.413404 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:40:02.413378    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbdba31a_15fa_4f08_af6b_cfb3d3c0bdb3.slice/crio-ca29cd1bfebc655a6359426608455c486157c98896a96febe8854e45692d4d1b WatchSource:0}: Error finding container ca29cd1bfebc655a6359426608455c486157c98896a96febe8854e45692d4d1b: Status 404 returned error can't find the container with id ca29cd1bfebc655a6359426608455c486157c98896a96febe8854e45692d4d1b
Nov 02 18:40:03.307041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:03.307001    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn" event={"ID":"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3","Type":"ContainerStarted","Data":"de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92"}
Nov 02 18:40:03.307438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:03.307046    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn" event={"ID":"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3","Type":"ContainerStarted","Data":"ca29cd1bfebc655a6359426608455c486157c98896a96febe8854e45692d4d1b"}
Nov 02 18:40:03.421971 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:03.421918    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn" podStartSLOduration=2.421905505 podStartE2EDuration="2.421905505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:40:03.418850845 +0000 UTC m=+4634.259456779" watchObservedRunningTime="2026-11-02 18:40:03.421905505 +0000 UTC m=+4634.262511439"
Nov 02 18:40:04.019745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:04.019702    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:04.257099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:04.257042    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:06.189183 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:06.189117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:08.648414 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:08.648367    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:08.827102 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:08.827063    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:09.275966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:09.275921    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:11.770565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:11.770530    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:12.019075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:12.019031    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:12.034034 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:12.033962    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:12.397599 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:12.397558    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:40:13.687547 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:13.687506    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:14.943227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:14.943192    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:15.142585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:15.142549    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:15.382365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:15.382332    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:17.736938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:17.736904    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:18.458704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:18.458662    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:20.443351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:20.443319    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:20.643587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:20.643559    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:20.922881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:20.922843    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:21.667343 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:21.667309    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:21.859923 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:21.859887    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:23.186450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:23.186404    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:26.839165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:26.839129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:27.025379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:27.025342    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:27.859723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:27.859680    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:29.061095 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:40:29.061052    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:40:32.492972 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:32.492931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:37.362872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:37.362833    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:37.695132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:37.695008    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:38.077127 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:38.077029    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:38.436330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:38.436294    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:38.788359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:38.788258    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:39.125170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:39.125132    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:39.457981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:39.457905    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:39.771669 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:39.771570    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:40.093865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:40.093785    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:41.685243 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:41.685187    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:42.009222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:42.009117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:42.963575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:42.963542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:44.570257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:44.570223    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:46.697109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:46.697041    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:51.319507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:51.319476    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:56.010285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:40:56.010251    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:40:59.058885 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:40:59.058846    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:41:00.695113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:00.695079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:01.897168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:01.897100    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:05.394672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:05.394602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:05.709420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:05.709326    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:06.488458 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:06.488427    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:07.541866 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:07.541834    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:10.071659 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:10.071617    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:11.608500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:11.608466    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:14.715884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:14.715855    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:16.415547 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.415509    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"]
Nov 02 18:41:16.419745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.419725    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.472811 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.472772    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.472811 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.472811    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.473227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.473208    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.473277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.473246    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fldjb\" (UniqueName: \"kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.473328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.473295    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.473328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.473320    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.474404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.474380    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"kube-root-ca.crt\""
Nov 02 18:41:16.474592 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.474493    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"openshift-service-ca.crt\""
Nov 02 18:41:16.474777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.474596    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"sub-test-model-413a4e-kserve-self-signed-certs\""
Nov 02 18:41:16.474990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.474975    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-413a4e\"/\"default-dockercfg-qrjpr\""
Nov 02 18:41:16.481232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.481202    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"]
Nov 02 18:41:16.501277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.501220    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:16.574620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.574590    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.574860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.574666    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.574860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.574783    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.574860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.574809    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.574860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.574847    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.575115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.574871    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fldjb\" (UniqueName: \"kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.575316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.575267    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.575478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.575381    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.575478 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.575464    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.577419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.577395    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.578091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.578068    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.631641 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.631585    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fldjb\" (UniqueName: \"kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb\") pod \"sub-test-model-413a4e-kserve-646b84db85-q6ts6\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") " pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:16.732221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:16.732119    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:17.127651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:17.127608    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"]
Nov 02 18:41:17.128595 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:41:17.128568    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2f06102_0149_4901_aaa5_e13243f51838.slice/crio-3dd3742cda7d849518e0e07d378c1c9a07828a097b4ab0ffeaca6d7d898359e6 WatchSource:0}: Error finding container 3dd3742cda7d849518e0e07d378c1c9a07828a097b4ab0ffeaca6d7d898359e6: Status 404 returned error can't find the container with id 3dd3742cda7d849518e0e07d378c1c9a07828a097b4ab0ffeaca6d7d898359e6
Nov 02 18:41:17.927124 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:17.927083    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6" event={"ID":"d2f06102-0149-4901-aaa5-e13243f51838","Type":"ContainerStarted","Data":"4ba21da7d79df2a003386a5515d893d2116ebd7483455189dfe515e018d27092"}
Nov 02 18:41:17.927521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:17.927137    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6" event={"ID":"d2f06102-0149-4901-aaa5-e13243f51838","Type":"ContainerStarted","Data":"3dd3742cda7d849518e0e07d378c1c9a07828a097b4ab0ffeaca6d7d898359e6"}
Nov 02 18:41:18.041354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:18.041295    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6" podStartSLOduration=2.041281205 podStartE2EDuration="2.041281205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:18.039155045 +0000 UTC m=+4708.879760982" watchObservedRunningTime="2026-11-02 18:41:18.041281205 +0000 UTC m=+4708.881887139"
Nov 02 18:41:18.732833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:18.732804    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:19.396609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:19.396562    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:24.100424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:24.100386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:26.732238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:26.732207    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:26.744753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:26.744729    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:27.004467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:27.004389    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:41:28.760826 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:28.760786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:29.060592 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:41:29.060503    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:41:31.784125 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.784040    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"]
Nov 02 18:41:31.788559 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.788538    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:31.844004 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.843977    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:31.844880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.844858    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"]
Nov 02 18:41:31.889457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.889406    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"kube-root-ca.crt\""
Nov 02 18:41:31.891322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.891287    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"default-dockercfg-vnmtz\""
Nov 02 18:41:31.891497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.891346    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"openshift-service-ca.crt\""
Nov 02 18:41:31.891497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.891377    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-580a21\"/\"sub-test-model-580a21-kserve-self-signed-certs\""
Nov 02 18:41:31.959425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.959398    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:31.959597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.959439    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:31.959597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.959521    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:31.959597 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.959560    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:31.959729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.959694    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhs8\" (UniqueName: \"kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:31.959729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:31.959725    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061017    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061182    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061229    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061279    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061308    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061408    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhs8\" (UniqueName: \"kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061496    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.061852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.061824    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.062089 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.062067    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.064666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.064001    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.064666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.064558    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.118841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.118805    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhs8\" (UniqueName: \"kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8\") pod \"sub-test-model-580a21-kserve-654f89b8c-q5pwd\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") " pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.402028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.401993    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:32.782074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:32.782046    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"]
Nov 02 18:41:32.783334 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:41:32.783307    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab4a298_dba5_4162_b226_e89896848f14.slice/crio-cf8db434cce66d41e8f376b3877e51db32a9d20e65117d3d1759206cd8e2e40e WatchSource:0}: Error finding container cf8db434cce66d41e8f376b3877e51db32a9d20e65117d3d1759206cd8e2e40e: Status 404 returned error can't find the container with id cf8db434cce66d41e8f376b3877e51db32a9d20e65117d3d1759206cd8e2e40e
Nov 02 18:41:33.035763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:33.035680    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" event={"ID":"fab4a298-dba5-4162-b226-e89896848f14","Type":"ContainerStarted","Data":"0e0cc5a590ff704823e4e308176100f489c24a5e7feb9717b1f696f82815875b"}
Nov 02 18:41:33.035763 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:33.035730    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" event={"ID":"fab4a298-dba5-4162-b226-e89896848f14","Type":"ContainerStarted","Data":"cf8db434cce66d41e8f376b3877e51db32a9d20e65117d3d1759206cd8e2e40e"}
Nov 02 18:41:33.149388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:33.149339    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" podStartSLOduration=2.149326042 podStartE2EDuration="2.149326042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:33.148283116 +0000 UTC m=+4723.988889054" watchObservedRunningTime="2026-11-02 18:41:33.149326042 +0000 UTC m=+4723.989931976"
Nov 02 18:41:33.402582 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:33.402554    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:33.438423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:33.438390    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:38.131700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:38.131666    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:42.329251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:42.329212    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:42.402828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:42.402797    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:42.415503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:42.415473    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:42.848399 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:42.848356    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:43.118762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:43.118732    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:41:43.189952 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:43.189917    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:43.510339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:43.510249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:45.046849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:45.046811    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"]
Nov 02 18:41:45.095836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:45.095805    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fa407c-v6gfm"]
Nov 02 18:41:45.384313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:45.384279    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f948e1e6-7557-414e-aabd-f44b92178700" path="/var/lib/kubelet/pods/f948e1e6-7557-414e-aabd-f44b92178700/volumes"
Nov 02 18:41:47.634489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:47.634459    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:47.942247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:47.942138    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:48.264347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:48.264243    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:51.575282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:51.575241    4476 scope.go:122] "RemoveContainer" containerID="2af3554ca7ea9cd3fae49b45eace00d0ecbc844b1be45ad412860b6def3a6aea"
Nov 02 18:41:52.704949 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:52.704913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:57.375113 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:57.375080    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:58.152375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:58.152340    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:58.483122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:41:58.483025    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:41:59.060742 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:41:59.060691    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:42:01.430437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:01.430396    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:02.212190 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:02.212152    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:06.830875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:06.830836    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:07.157958 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:07.157912    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:11.623283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:11.623249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:16.314117 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:16.314079    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:21.005197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:21.005166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:25.654535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:25.654489    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:29.062678 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:42:29.062606    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:42:30.330546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:30.330511    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:34.980278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:34.980241    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:38.596536 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:38.596495    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:39.700860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:39.700820    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:44.416020 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:44.415988    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:49.105317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:49.105280    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:50.220975 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:50.220945    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:42:50.231196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:50.231171    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:42:51.606516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:51.606482    4476 scope.go:122] "RemoveContainer" containerID="e2c42b490aba9b2afba0af3f5d83b7113d0cc4dc353cc528d716bafa5c8cc9cf"
Nov 02 18:42:53.756127 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:53.756092    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:58.364659 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:42:58.364603    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:42:59.060845 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:42:59.060801    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:43:03.028611 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:03.028540    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:03.337908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:03.337825    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:04.028229 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.028199    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"]
Nov 02 18:43:04.032015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.031999    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.076056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.076022    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.076246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.076121    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.076246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.076175    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.076246 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.076225    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljmk\" (UniqueName: \"kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.076443 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.076265    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.076443 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.076365    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.092641 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.092595    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"]
Nov 02 18:43:04.127845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.127817    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:04.136394 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.136373    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"sub-test-model-c82472-kserve-self-signed-certs\""
Nov 02 18:43:04.136511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.136411    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"openshift-service-ca.crt\""
Nov 02 18:43:04.136511 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.136410    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"default-dockercfg-vjh7q\""
Nov 02 18:43:04.137135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.137123    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c82472\"/\"kube-root-ca.crt\""
Nov 02 18:43:04.177442 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177411    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljmk\" (UniqueName: \"kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.177655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177465    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.177655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177566    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.177814 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177704    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.177814 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177778    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.177937 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177824    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.178000 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.177960    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.178062 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.178041    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.178198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.178180    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.181610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.181582    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.181867 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.181845    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.234373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.234334    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljmk\" (UniqueName: \"kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk\") pod \"sub-test-model-c82472-kserve-65d97f4bbf-k7dqv\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") " pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.345358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.345210    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:04.735932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:04.735899    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"]
Nov 02 18:43:04.736736 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:43:04.736711    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod642a337b_9bbb_4202_bc14_57bfa7869568.slice/crio-44392f931c129ce243fd7db60a50634f80ffae3f2b7cedd673a3ebb82a91fd2b WatchSource:0}: Error finding container 44392f931c129ce243fd7db60a50634f80ffae3f2b7cedd673a3ebb82a91fd2b: Status 404 returned error can't find the container with id 44392f931c129ce243fd7db60a50634f80ffae3f2b7cedd673a3ebb82a91fd2b
Nov 02 18:43:05.721828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:05.721794    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv" event={"ID":"642a337b-9bbb-4202-bc14-57bfa7869568","Type":"ContainerStarted","Data":"9b1990ba0b7dadc1fd53eed3e5cc74abd48f6d05d8e8a323e571da969c91b76b"}
Nov 02 18:43:05.722225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:05.721837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv" event={"ID":"642a337b-9bbb-4202-bc14-57bfa7869568","Type":"ContainerStarted","Data":"44392f931c129ce243fd7db60a50634f80ffae3f2b7cedd673a3ebb82a91fd2b"}
Nov 02 18:43:05.835362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:05.835308    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv" podStartSLOduration=2.835292153 podStartE2EDuration="2.835292153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:05.833340865 +0000 UTC m=+4816.673946800" watchObservedRunningTime="2026-11-02 18:43:05.835292153 +0000 UTC m=+4816.675898087"
Nov 02 18:43:06.345345 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:06.345313    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:07.180498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:07.180458    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:07.928762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:07.928724    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:12.441017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:12.440979    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:12.821155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:12.820207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:14.345695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:14.345661    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:14.359072 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:14.359047    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:14.799925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:14.799883    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:43:17.396344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:17.396311    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:22.117193 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:22.117155    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:23.049058 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:23.049023    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"]
Nov 02 18:43:23.097766 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:23.097731    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f3e8628-jtfsh"]
Nov 02 18:43:23.384431 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:23.384384    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699a874b-ec31-4413-af94-e29840916ed1" path="/var/lib/kubelet/pods/699a874b-ec31-4413-af94-e29840916ed1/volumes"
Nov 02 18:43:26.801787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:26.801753    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:29.062196 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:43:29.062156    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:43:31.749306 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:31.749139    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:36.423152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:36.423119    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:37.485989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:37.485945    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:37.820106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:37.820020    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:41.163845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:41.163805    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:45.954213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:45.954178    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:50.704782 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:50.704748    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:55.391459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:43:55.391422    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:43:59.060492 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:43:59.060456    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:44:00.045985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:00.045942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:04.686575 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:04.686542    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:07.520723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:07.520676    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:09.342519 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:09.342485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:14.059162 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:14.059129    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:18.653418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:18.653379    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:23.301862 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:23.301826    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:28.091261 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:28.091221    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:29.060973 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:44:29.060930    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:44:31.424351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:31.424264    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:32.354460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:32.354424    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:32.683509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:32.683480    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:33.025423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:33.025260    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:33.404883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:33.404849    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:37.680195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:37.680153    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:38.938022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:38.937992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:42.407809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:42.407769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:47.105383 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:47.105336    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:51.644279 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:51.644239    4476 scope.go:122] "RemoveContainer" containerID="9b6cec547dfdce53915f6c5cc1279037ba3605e3edd35cd80affba41cc29e35d"
Nov 02 18:44:51.741026 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:51.740976    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:56.516890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:44:56.516857    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:44:59.061008 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:44:59.060963    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:45:00.173609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.173574    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"]
Nov 02 18:45:00.179178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.179153    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"]
Nov 02 18:45:00.179352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.179285    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.183377 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.183353    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.204822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.204793    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"]
Nov 02 18:45:00.208857 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.208835    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.220898 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.220871    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"]
Nov 02 18:45:00.224830 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.224809    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"]
Nov 02 18:45:00.224975 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.224941    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.228584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.228566    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.238388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.238360    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"]
Nov 02 18:45:00.242765 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.242746    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.275031 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.274996    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"]
Nov 02 18:45:00.278726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.278704    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"]
Nov 02 18:45:00.280114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280095    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4dg6\" (UniqueName: \"kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.280203 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280137    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.280203 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280157    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ncf\" (UniqueName: \"kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.280288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280227    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htzt\" (UniqueName: \"kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt\") pod \"maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.280330 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280314    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.280372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280341    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.280372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280366    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.280508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280491    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.280548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280516    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggcp6\" (UniqueName: \"kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.280594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.280547    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfg4m\" (UniqueName: \"kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.326144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.326103    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"]
Nov 02 18:45:00.327149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.327121    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:45:00.381849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.381809    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.382957 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.382931    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.383876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.383858    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.384744 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.384722    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.384925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.384907    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.390422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.384602    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.390422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.383796    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.390422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.382850    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.390422 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.388365    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"]
Nov 02 18:45:00.390741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.390561    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.390875 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.390856    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggcp6\" (UniqueName: \"kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.391322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.391300    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfg4m\" (UniqueName: \"kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.392466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.392442    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7hv\" (UniqueName: \"kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.392620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.392602    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4dg6\" (UniqueName: \"kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.392796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.392778    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.393067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.393039    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ncf\" (UniqueName: \"kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.393237 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.393215    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htzt\" (UniqueName: \"kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt\") pod \"maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.393463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.393446    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"]
Nov 02 18:45:00.394370 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.394349    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.398103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.398085    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"]
Nov 02 18:45:00.449605 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.449512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggcp6\" (UniqueName: \"kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.449795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.449675    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htzt\" (UniqueName: \"kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt\") pod \"maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.449795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.449782    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfg4m\" (UniqueName: \"kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.450354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.450330    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4dg6\" (UniqueName: \"kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.450836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.450813    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ncf\" (UniqueName: \"kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.494305 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.494267    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7hv\" (UniqueName: \"kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.494529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.494396    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.494529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.494449    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:00.495066 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.495049    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.501392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.501371    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:00.523880 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.523441    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:00.540845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.540821    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:00.549039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.548993    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:00.553108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.552844    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7hv\" (UniqueName: \"kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.559166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.559010    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:00.905215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.905183    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"]
Nov 02 18:45:00.906446 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:45:00.906412    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e3299b_30b4_43da_a27c_60c85b034f4e.slice/crio-aed9986a63b8e5a7f15e8bbb8dc4ac640e7cb24acc33288b2a545f148454c3e2 WatchSource:0}: Error finding container aed9986a63b8e5a7f15e8bbb8dc4ac640e7cb24acc33288b2a545f148454c3e2: Status 404 returned error can't find the container with id aed9986a63b8e5a7f15e8bbb8dc4ac640e7cb24acc33288b2a545f148454c3e2
Nov 02 18:45:00.956474 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:00.956445    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"]
Nov 02 18:45:01.027202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.027169    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"]
Nov 02 18:45:01.032845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.032816    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"]
Nov 02 18:45:01.036417 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.036393    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"]
Nov 02 18:45:01.040926 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:45:01.040900    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e372001_8a06_47f1_ba3b_17ee0f102102.slice/crio-6acc1983a42585faf1dfcd1dbbba53f02d14c5029f662ead2b6f264eab4e3c27 WatchSource:0}: Error finding container 6acc1983a42585faf1dfcd1dbbba53f02d14c5029f662ead2b6f264eab4e3c27: Status 404 returned error can't find the container with id 6acc1983a42585faf1dfcd1dbbba53f02d14c5029f662ead2b6f264eab4e3c27
Nov 02 18:45:01.041669 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:45:01.041611    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e24920_a5e4_46a2_b782_3eb5d22f3b1e.slice/crio-465fbead6128008355176a2462e64834e566f7d501501c08d9a87759d0c58b76 WatchSource:0}: Error finding container 465fbead6128008355176a2462e64834e566f7d501501c08d9a87759d0c58b76: Status 404 returned error can't find the container with id 465fbead6128008355176a2462e64834e566f7d501501c08d9a87759d0c58b76
Nov 02 18:45:01.044881 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:45:01.044844    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71919475_10bb_4d20_a936_30e2a871b69f.slice/crio-f9521716b5046d44f8c43aebbbfa6212b4acf50e8acf28aea0a52027cbadfc34 WatchSource:0}: Error finding container f9521716b5046d44f8c43aebbbfa6212b4acf50e8acf28aea0a52027cbadfc34: Status 404 returned error can't find the container with id f9521716b5046d44f8c43aebbbfa6212b4acf50e8acf28aea0a52027cbadfc34
Nov 02 18:45:01.050655 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:45:01.047713    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d58be1_880c_4fc2_8104_ddc4d02a3adf.slice/crio-043ab9600168d95b6d04c12d9b90d571d833c76ae2d176b056d09fa0df3f5bc4 WatchSource:0}: Error finding container 043ab9600168d95b6d04c12d9b90d571d833c76ae2d176b056d09fa0df3f5bc4: Status 404 returned error can't find the container with id 043ab9600168d95b6d04c12d9b90d571d833c76ae2d176b056d09fa0df3f5bc4
Nov 02 18:45:01.120640 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.120596    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"]
Nov 02 18:45:01.121671 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:45:01.121618    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80674c25_365d_4314_8b60_8e393f630983.slice/crio-c6a1bb7b55e9fa186360af57eea0825a64053042beda5e1abc02860724398800 WatchSource:0}: Error finding container c6a1bb7b55e9fa186360af57eea0825a64053042beda5e1abc02860724398800: Status 404 returned error can't find the container with id c6a1bb7b55e9fa186360af57eea0825a64053042beda5e1abc02860724398800
Nov 02 18:45:01.306350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.306318    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:01.633813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.633777    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerStarted","Data":"dd7694cdd6dc42939ecb51e84014369f30915de74ac77d683702ceb6bad2027c"}
Nov 02 18:45:01.633994 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.633846    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerStarted","Data":"043ab9600168d95b6d04c12d9b90d571d833c76ae2d176b056d09fa0df3f5bc4"}
Nov 02 18:45:01.635403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.635371    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerStarted","Data":"c6a1bb7b55e9fa186360af57eea0825a64053042beda5e1abc02860724398800"}
Nov 02 18:45:01.636673 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.636646    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" event={"ID":"2e372001-8a06-47f1-ba3b-17ee0f102102","Type":"ContainerStarted","Data":"6acc1983a42585faf1dfcd1dbbba53f02d14c5029f662ead2b6f264eab4e3c27"}
Nov 02 18:45:01.640993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.638226    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerStarted","Data":"a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9"}
Nov 02 18:45:01.640993 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.638296    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerStarted","Data":"f9521716b5046d44f8c43aebbbfa6212b4acf50e8acf28aea0a52027cbadfc34"}
Nov 02 18:45:01.643429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.642523    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerStarted","Data":"876d5440372131c67433d48d681ff9177674edb74c7f213bc79564991dbd8e63"}
Nov 02 18:45:01.643429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.642585    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerStarted","Data":"465fbead6128008355176a2462e64834e566f7d501501c08d9a87759d0c58b76"}
Nov 02 18:45:01.644738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.644707    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerStarted","Data":"7b98273a8626ced774550f29e4991f0a7b520a440d8573359ea3ccae176ce589"}
Nov 02 18:45:01.644869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.644758    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerStarted","Data":"aed9986a63b8e5a7f15e8bbb8dc4ac640e7cb24acc33288b2a545f148454c3e2"}
Nov 02 18:45:01.747234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.747169    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" podStartSLOduration=1.74714826 podStartE2EDuration="1.74714826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:01.744734747 +0000 UTC m=+4932.585340683" watchObservedRunningTime="2026-11-02 18:45:01.74714826 +0000 UTC m=+4932.587754195"
Nov 02 18:45:01.856042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.855918    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" podStartSLOduration=1.855899393 podStartE2EDuration="1.855899393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:01.854836696 +0000 UTC m=+4932.695442629" watchObservedRunningTime="2026-11-02 18:45:01.855899393 +0000 UTC m=+4932.696505329"
Nov 02 18:45:01.966164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:01.966092    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" podStartSLOduration=1.9656828050000001 podStartE2EDuration="1.965682805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:01.965122546 +0000 UTC m=+4932.805728481" watchObservedRunningTime="2026-11-02 18:45:01.965682805 +0000 UTC m=+4932.806288740"
Nov 02 18:45:02.078372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.078305    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" podStartSLOduration=2.078284741 podStartE2EDuration="2.078284741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.075272867 +0000 UTC m=+4932.915878799" watchObservedRunningTime="2026-11-02 18:45:02.078284741 +0000 UTC m=+4932.918890675"
Nov 02 18:45:02.285562 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:02.285523    4476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71919475_10bb_4d20_a936_30e2a871b69f.slice/crio-conmon-a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71919475_10bb_4d20_a936_30e2a871b69f.slice/crio-a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:45:02.399953 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.399863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:02.654755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.654665    4476 generic.go:356] "Generic (PLEG): container finished" podID="71919475-10bb-4d20-a936-30e2a871b69f" containerID="a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9" exitCode=60
Nov 02 18:45:02.654927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.654779    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerDied","Data":"a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9"}
Nov 02 18:45:02.655123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.655105    4476 scope.go:122] "RemoveContainer" containerID="a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9"
Nov 02 18:45:02.657253 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.657227    4476 generic.go:356] "Generic (PLEG): container finished" podID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerID="876d5440372131c67433d48d681ff9177674edb74c7f213bc79564991dbd8e63" exitCode=60
Nov 02 18:45:02.657373 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.657317    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerDied","Data":"876d5440372131c67433d48d681ff9177674edb74c7f213bc79564991dbd8e63"}
Nov 02 18:45:02.657587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.657570    4476 scope.go:122] "RemoveContainer" containerID="876d5440372131c67433d48d681ff9177674edb74c7f213bc79564991dbd8e63"
Nov 02 18:45:02.659354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.659328    4476 generic.go:356] "Generic (PLEG): container finished" podID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerID="7b98273a8626ced774550f29e4991f0a7b520a440d8573359ea3ccae176ce589" exitCode=60
Nov 02 18:45:02.659441 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.659413    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerDied","Data":"7b98273a8626ced774550f29e4991f0a7b520a440d8573359ea3ccae176ce589"}
Nov 02 18:45:02.660107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.660089    4476 scope.go:122] "RemoveContainer" containerID="7b98273a8626ced774550f29e4991f0a7b520a440d8573359ea3ccae176ce589"
Nov 02 18:45:02.661408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.661385    4476 generic.go:356] "Generic (PLEG): container finished" podID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerID="dd7694cdd6dc42939ecb51e84014369f30915de74ac77d683702ceb6bad2027c" exitCode=60
Nov 02 18:45:02.661499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.661469    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerDied","Data":"dd7694cdd6dc42939ecb51e84014369f30915de74ac77d683702ceb6bad2027c"}
Nov 02 18:45:02.661814 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.661778    4476 scope.go:122] "RemoveContainer" containerID="dd7694cdd6dc42939ecb51e84014369f30915de74ac77d683702ceb6bad2027c"
Nov 02 18:45:02.663695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.663673    4476 generic.go:356] "Generic (PLEG): container finished" podID="80674c25-365d-4314-8b60-8e393f630983" containerID="3621770da7d0f1ad07ddbe354e92764f085cd8f12e10e9ef2cd6ee964817b2c7" exitCode=60
Nov 02 18:45:02.663961 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.663925    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerDied","Data":"3621770da7d0f1ad07ddbe354e92764f085cd8f12e10e9ef2cd6ee964817b2c7"}
Nov 02 18:45:02.664080 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.664051    4476 scope.go:122] "RemoveContainer" containerID="3621770da7d0f1ad07ddbe354e92764f085cd8f12e10e9ef2cd6ee964817b2c7"
Nov 02 18:45:02.665759 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.665734    4476 generic.go:356] "Generic (PLEG): container finished" podID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerID="9efea860723daecf11e80b677845f1f2485bf2e438c6a273d08ca30eeea1b599" exitCode=60
Nov 02 18:45:02.665890 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.665787    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" event={"ID":"2e372001-8a06-47f1-ba3b-17ee0f102102","Type":"ContainerDied","Data":"9efea860723daecf11e80b677845f1f2485bf2e438c6a273d08ca30eeea1b599"}
Nov 02 18:45:02.666030 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:02.666013    4476 scope.go:122] "RemoveContainer" containerID="9efea860723daecf11e80b677845f1f2485bf2e438c6a273d08ca30eeea1b599"
Nov 02 18:45:03.677854 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.677810    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerStarted","Data":"c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6"}
Nov 02 18:45:03.680318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.680288    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerStarted","Data":"d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074"}
Nov 02 18:45:03.682448 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.682424    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" event={"ID":"2e372001-8a06-47f1-ba3b-17ee0f102102","Type":"ContainerStarted","Data":"9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208"}
Nov 02 18:45:03.684787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.684761    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerStarted","Data":"b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa"}
Nov 02 18:45:03.687013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.686990    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerStarted","Data":"96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd"}
Nov 02 18:45:03.688942 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.688916    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerStarted","Data":"b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52"}
Nov 02 18:45:03.908344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:03.908282    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" podStartSLOduration=3.908268856 podStartE2EDuration="3.908268856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:03.90389124 +0000 UTC m=+4934.744497173" watchObservedRunningTime="2026-11-02 18:45:03.908268856 +0000 UTC m=+4934.748874790"
Nov 02 18:45:04.015907 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.015859    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" podStartSLOduration=4.015845249 podStartE2EDuration="4.015845249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:04.014449334 +0000 UTC m=+4934.855055268" watchObservedRunningTime="2026-11-02 18:45:04.015845249 +0000 UTC m=+4934.856451183"
Nov 02 18:45:04.698098 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.698061    4476 generic.go:356] "Generic (PLEG): container finished" podID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerID="c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6" exitCode=60
Nov 02 18:45:04.698577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.698153    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerDied","Data":"c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6"}
Nov 02 18:45:04.698577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.698224    4476 scope.go:122] "RemoveContainer" containerID="dd7694cdd6dc42939ecb51e84014369f30915de74ac77d683702ceb6bad2027c"
Nov 02 18:45:04.698577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.698436    4476 scope.go:122] "RemoveContainer" containerID="c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6"
Nov 02 18:45:04.698800 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:04.698780    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm_odh-ai-gateway-infra(49d58be1-880c-4fc2-8104-ddc4d02a3adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf"
Nov 02 18:45:04.700348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.700328    4476 generic.go:356] "Generic (PLEG): container finished" podID="80674c25-365d-4314-8b60-8e393f630983" containerID="d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074" exitCode=60
Nov 02 18:45:04.700469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.700411    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerDied","Data":"d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074"}
Nov 02 18:45:04.700686 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.700668    4476 scope.go:122] "RemoveContainer" containerID="d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074"
Nov 02 18:45:04.700993 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:04.700974    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh_odh-ai-gateway-infra(80674c25-365d-4314-8b60-8e393f630983)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" podUID="80674c25-365d-4314-8b60-8e393f630983"
Nov 02 18:45:04.702460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.702439    4476 generic.go:356] "Generic (PLEG): container finished" podID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerID="9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208" exitCode=60
Nov 02 18:45:04.702578 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.702513    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" event={"ID":"2e372001-8a06-47f1-ba3b-17ee0f102102","Type":"ContainerDied","Data":"9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208"}
Nov 02 18:45:04.702758 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.702742    4476 scope.go:122] "RemoveContainer" containerID="9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208"
Nov 02 18:45:04.703024 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:04.703003    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt_odh-ai-gateway-infra(2e372001-8a06-47f1-ba3b-17ee0f102102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102"
Nov 02 18:45:04.704520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.704501    4476 generic.go:356] "Generic (PLEG): container finished" podID="71919475-10bb-4d20-a936-30e2a871b69f" containerID="b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa" exitCode=60
Nov 02 18:45:04.704602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.704587    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerDied","Data":"b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa"}
Nov 02 18:45:04.704822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.704805    4476 scope.go:122] "RemoveContainer" containerID="b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa"
Nov 02 18:45:04.705080 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:04.705058    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv_odh-ai-gateway-infra(71919475-10bb-4d20-a936-30e2a871b69f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" podUID="71919475-10bb-4d20-a936-30e2a871b69f"
Nov 02 18:45:04.706614 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.706593    4476 generic.go:356] "Generic (PLEG): container finished" podID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerID="96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd" exitCode=60
Nov 02 18:45:04.706742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.706661    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerDied","Data":"96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd"}
Nov 02 18:45:04.706881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.706868    4476 scope.go:122] "RemoveContainer" containerID="96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd"
Nov 02 18:45:04.707111 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:04.707091    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq_odh-ai-gateway-infra(55e24920-a5e4-46a2-b782-3eb5d22f3b1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e"
Nov 02 18:45:04.708501 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.708482    4476 generic.go:356] "Generic (PLEG): container finished" podID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerID="b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52" exitCode=60
Nov 02 18:45:04.708585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.708527    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerDied","Data":"b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52"}
Nov 02 18:45:04.708743 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.708729    4476 scope.go:122] "RemoveContainer" containerID="b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52"
Nov 02 18:45:04.709007 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:04.708987    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6_odh-ai-gateway-infra(03e3299b-30b4-43da-a27c-60c85b034f4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e"
Nov 02 18:45:04.711604 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.711588    4476 scope.go:122] "RemoveContainer" containerID="3621770da7d0f1ad07ddbe354e92764f085cd8f12e10e9ef2cd6ee964817b2c7"
Nov 02 18:45:04.723671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.723617    4476 scope.go:122] "RemoveContainer" containerID="9efea860723daecf11e80b677845f1f2485bf2e438c6a273d08ca30eeea1b599"
Nov 02 18:45:04.735793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.735775    4476 scope.go:122] "RemoveContainer" containerID="a9aee5f23e0ccfa4bf41e3a820bef9b49b9d84a7c76a9f7dfbc2677cee25b8d9"
Nov 02 18:45:04.748806 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.748779    4476 scope.go:122] "RemoveContainer" containerID="876d5440372131c67433d48d681ff9177674edb74c7f213bc79564991dbd8e63"
Nov 02 18:45:04.761964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:04.761939    4476 scope.go:122] "RemoveContainer" containerID="7b98273a8626ced774550f29e4991f0a7b520a440d8573359ea3ccae176ce589"
Nov 02 18:45:05.723756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:05.723726    4476 scope.go:122] "RemoveContainer" containerID="9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208"
Nov 02 18:45:05.724209 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:05.724108    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt_odh-ai-gateway-infra(2e372001-8a06-47f1-ba3b-17ee0f102102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102"
Nov 02 18:45:05.725425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:05.725399    4476 scope.go:122] "RemoveContainer" containerID="b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa"
Nov 02 18:45:05.725747 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:05.725724    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv_odh-ai-gateway-infra(71919475-10bb-4d20-a936-30e2a871b69f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" podUID="71919475-10bb-4d20-a936-30e2a871b69f"
Nov 02 18:45:05.727154 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:05.727130    4476 scope.go:122] "RemoveContainer" containerID="96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd"
Nov 02 18:45:05.727461 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:05.727432    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq_odh-ai-gateway-infra(55e24920-a5e4-46a2-b782-3eb5d22f3b1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e"
Nov 02 18:45:05.728884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:05.728862    4476 scope.go:122] "RemoveContainer" containerID="b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52"
Nov 02 18:45:05.729155 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:05.729129    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6_odh-ai-gateway-infra(03e3299b-30b4-43da-a27c-60c85b034f4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e"
Nov 02 18:45:05.730692 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:05.730673    4476 scope.go:122] "RemoveContainer" containerID="c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6"
Nov 02 18:45:05.730942 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:05.730924    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm_odh-ai-gateway-infra(49d58be1-880c-4fc2-8104-ddc4d02a3adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf"
Nov 02 18:45:05.732188 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:05.732171    4476 scope.go:122] "RemoveContainer" containerID="d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074"
Nov 02 18:45:05.732403 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:05.732386    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh_odh-ai-gateway-infra(80674c25-365d-4314-8b60-8e393f630983)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" podUID="80674c25-365d-4314-8b60-8e393f630983"
Nov 02 18:45:06.002043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:06.001928    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:07.891679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:07.891647    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:10.669233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:10.669196    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:14.128525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:14.128478    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:15.304388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:15.304354    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:15.388320 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:15.388276    4476 scope.go:122] "RemoveContainer" containerID="c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6"
Nov 02 18:45:15.391362 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:15.391335    4476 scope.go:122] "RemoveContainer" containerID="96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd"
Nov 02 18:45:15.391786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:15.391762    4476 scope.go:122] "RemoveContainer" containerID="b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52"
Nov 02 18:45:15.807720 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:15.807682    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerStarted","Data":"a669c567487418f1aff834c238bab8521b7405bf19d07d5855d424dcc015265c"}
Nov 02 18:45:16.817406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.817319    4476 generic.go:356] "Generic (PLEG): container finished" podID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerID="eeed0d26abaea545d9077d2387f298875b81a82c459e80c5c8da417e3067efaf" exitCode=60
Nov 02 18:45:16.817406 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.817387    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerDied","Data":"eeed0d26abaea545d9077d2387f298875b81a82c459e80c5c8da417e3067efaf"}
Nov 02 18:45:16.817986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.817455    4476 scope.go:122] "RemoveContainer" containerID="c170d91cd0979e2aac512cc5c6f41e0deab801899d454ad04e6ac45f5e501be6"
Nov 02 18:45:16.817986 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.817922    4476 scope.go:122] "RemoveContainer" containerID="eeed0d26abaea545d9077d2387f298875b81a82c459e80c5c8da417e3067efaf"
Nov 02 18:45:16.818261 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:16.818236    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm_odh-ai-gateway-infra(49d58be1-880c-4fc2-8104-ddc4d02a3adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf"
Nov 02 18:45:16.820215 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.820177    4476 generic.go:356] "Generic (PLEG): container finished" podID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerID="2ee5b443786d68b45104929f200fc50235465737798fc0446d966e295bff3269" exitCode=60
Nov 02 18:45:16.820341 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.820252    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerDied","Data":"2ee5b443786d68b45104929f200fc50235465737798fc0446d966e295bff3269"}
Nov 02 18:45:16.820742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.820722    4476 scope.go:122] "RemoveContainer" containerID="2ee5b443786d68b45104929f200fc50235465737798fc0446d966e295bff3269"
Nov 02 18:45:16.821299 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:16.821003    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq_odh-ai-gateway-infra(55e24920-a5e4-46a2-b782-3eb5d22f3b1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e"
Nov 02 18:45:16.822909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.822883    4476 generic.go:356] "Generic (PLEG): container finished" podID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerID="a669c567487418f1aff834c238bab8521b7405bf19d07d5855d424dcc015265c" exitCode=60
Nov 02 18:45:16.823018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.822965    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerDied","Data":"a669c567487418f1aff834c238bab8521b7405bf19d07d5855d424dcc015265c"}
Nov 02 18:45:16.823289 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.823274    4476 scope.go:122] "RemoveContainer" containerID="a669c567487418f1aff834c238bab8521b7405bf19d07d5855d424dcc015265c"
Nov 02 18:45:16.823527 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:16.823507    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6_odh-ai-gateway-infra(03e3299b-30b4-43da-a27c-60c85b034f4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e"
Nov 02 18:45:16.832064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.832034    4476 scope.go:122] "RemoveContainer" containerID="96705141e7a66785e7e3123eff607d412ae8ed2d5eb6b3ed20b39cdb35699bfd"
Nov 02 18:45:16.853136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.853111    4476 scope.go:122] "RemoveContainer" containerID="b90b228c2836a94e470eb772b29f70bf46644cfba1d375cdcdd288a61f53fc52"
Nov 02 18:45:16.971171 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:16.971134    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"]
Nov 02 18:45:17.376899 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:17.376865    4476 scope.go:122] "RemoveContainer" containerID="b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa"
Nov 02 18:45:17.377122 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:17.377058    4476 scope.go:122] "RemoveContainer" containerID="d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074"
Nov 02 18:45:17.377452 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:17.377428    4476 scope.go:122] "RemoveContainer" containerID="9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208"
Nov 02 18:45:17.832290 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:17.832248    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerStarted","Data":"97452782cadf912113dbedb66a6c7433658277d28f4134b20d719dd3768f76d7"}
Nov 02 18:45:17.839504 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:17.839454    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerStarted","Data":"52f9311f4fbf486d5c34f26791f297043fbec474944162a9bfcd946cce5eb16c"}
Nov 02 18:45:17.949093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:17.949061    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"]
Nov 02 18:45:18.035753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.035724    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:18.130898 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.130871    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:18.133039 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.133021    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf\" (UniqueName: \"kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf\") pod \"03e3299b-30b4-43da-a27c-60c85b034f4e\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") "
Nov 02 18:45:18.133164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.133149    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca\") pod \"03e3299b-30b4-43da-a27c-60c85b034f4e\" (UID: \"03e3299b-30b4-43da-a27c-60c85b034f4e\") "
Nov 02 18:45:18.133467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.133442    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca" pod "03e3299b-30b4-43da-a27c-60c85b034f4e" (UID: "03e3299b-30b4-43da-a27c-60c85b034f4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:18.135170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.135150    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf" pod "03e3299b-30b4-43da-a27c-60c85b034f4e" (UID: "03e3299b-30b4-43da-a27c-60c85b034f4e"). InnerVolumeSpecName "kube-api-access-22ncf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:18.169221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.169184    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"]
Nov 02 18:45:18.234054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.233971    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv\" (UniqueName: \"kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv\") pod \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") "
Nov 02 18:45:18.234054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.234008    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca\") pod \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\" (UID: \"49d58be1-880c-4fc2-8104-ddc4d02a3adf\") "
Nov 02 18:45:18.234400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.234375    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e3299b-30b4-43da-a27c-60c85b034f4e-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:18.234517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.234404    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ncf\" (UniqueName: \"kubernetes.io/projected/03e3299b-30b4-43da-a27c-60c85b034f4e-kube-api-access-22ncf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:18.234517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.234435    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca" pod "49d58be1-880c-4fc2-8104-ddc4d02a3adf" (UID: "49d58be1-880c-4fc2-8104-ddc4d02a3adf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:18.236488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.236461    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv" pod "49d58be1-880c-4fc2-8104-ddc4d02a3adf" (UID: "49d58be1-880c-4fc2-8104-ddc4d02a3adf"). InnerVolumeSpecName "kube-api-access-lk7hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:18.318935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.318906    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:18.336134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.336091    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7hv\" (UniqueName: \"kubernetes.io/projected/49d58be1-880c-4fc2-8104-ddc4d02a3adf-kube-api-access-lk7hv\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:18.336134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.336124    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d58be1-880c-4fc2-8104-ddc4d02a3adf-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:18.436594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.436552    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m\" (UniqueName: \"kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m\") pod \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") "
Nov 02 18:45:18.436827 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.436805    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca\") pod \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\" (UID: \"55e24920-a5e4-46a2-b782-3eb5d22f3b1e\") "
Nov 02 18:45:18.437254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.437221    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca" pod "55e24920-a5e4-46a2-b782-3eb5d22f3b1e" (UID: "55e24920-a5e4-46a2-b782-3eb5d22f3b1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:18.437412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.437394    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:18.438988 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.438962    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m" pod "55e24920-a5e4-46a2-b782-3eb5d22f3b1e" (UID: "55e24920-a5e4-46a2-b782-3eb5d22f3b1e"). InnerVolumeSpecName "kube-api-access-bfg4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:18.538438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.538341    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfg4m\" (UniqueName: \"kubernetes.io/projected/55e24920-a5e4-46a2-b782-3eb5d22f3b1e-kube-api-access-bfg4m\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:18.849286 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.849197    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"
Nov 02 18:45:18.849286 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.849219    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6" event={"ID":"03e3299b-30b4-43da-a27c-60c85b034f4e","Type":"ContainerDied","Data":"aed9986a63b8e5a7f15e8bbb8dc4ac640e7cb24acc33288b2a545f148454c3e2"}
Nov 02 18:45:18.849286 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.849278    4476 scope.go:122] "RemoveContainer" containerID="a669c567487418f1aff834c238bab8521b7405bf19d07d5855d424dcc015265c"
Nov 02 18:45:18.856339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.856305    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm" event={"ID":"49d58be1-880c-4fc2-8104-ddc4d02a3adf","Type":"ContainerDied","Data":"043ab9600168d95b6d04c12d9b90d571d833c76ae2d176b056d09fa0df3f5bc4"}
Nov 02 18:45:18.856500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.856484    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"
Nov 02 18:45:18.867879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.867836    4476 generic.go:356] "Generic (PLEG): container finished" podID="80674c25-365d-4314-8b60-8e393f630983" containerID="52f9311f4fbf486d5c34f26791f297043fbec474944162a9bfcd946cce5eb16c" exitCode=60
Nov 02 18:45:18.868043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.867998    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerDied","Data":"52f9311f4fbf486d5c34f26791f297043fbec474944162a9bfcd946cce5eb16c"}
Nov 02 18:45:18.871219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.868481    4476 scope.go:122] "RemoveContainer" containerID="52f9311f4fbf486d5c34f26791f297043fbec474944162a9bfcd946cce5eb16c"
Nov 02 18:45:18.871219 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:18.870231    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh_odh-ai-gateway-infra(80674c25-365d-4314-8b60-8e393f630983)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" podUID="80674c25-365d-4314-8b60-8e393f630983"
Nov 02 18:45:18.887281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.887171    4476 scope.go:122] "RemoveContainer" containerID="eeed0d26abaea545d9077d2387f298875b81a82c459e80c5c8da417e3067efaf"
Nov 02 18:45:18.888106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.888077    4476 generic.go:356] "Generic (PLEG): container finished" podID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerID="6a2db4b4f3243960a2347a7bdc4cca4f91dff896556e511b5d4f52be0dfeda29" exitCode=60
Nov 02 18:45:18.888238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.888195    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" event={"ID":"2e372001-8a06-47f1-ba3b-17ee0f102102","Type":"ContainerDied","Data":"6a2db4b4f3243960a2347a7bdc4cca4f91dff896556e511b5d4f52be0dfeda29"}
Nov 02 18:45:18.888798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.888618    4476 scope.go:122] "RemoveContainer" containerID="6a2db4b4f3243960a2347a7bdc4cca4f91dff896556e511b5d4f52be0dfeda29"
Nov 02 18:45:18.889047 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:18.889031    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt_odh-ai-gateway-infra(2e372001-8a06-47f1-ba3b-17ee0f102102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102"
Nov 02 18:45:18.890909 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.890887    4476 generic.go:356] "Generic (PLEG): container finished" podID="71919475-10bb-4d20-a936-30e2a871b69f" containerID="97452782cadf912113dbedb66a6c7433658277d28f4134b20d719dd3768f76d7" exitCode=60
Nov 02 18:45:18.891015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.890911    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerDied","Data":"97452782cadf912113dbedb66a6c7433658277d28f4134b20d719dd3768f76d7"}
Nov 02 18:45:18.891264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.891246    4476 scope.go:122] "RemoveContainer" containerID="97452782cadf912113dbedb66a6c7433658277d28f4134b20d719dd3768f76d7"
Nov 02 18:45:18.891511 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:18.891484    4476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv_odh-ai-gateway-infra(71919475-10bb-4d20-a936-30e2a871b69f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" podUID="71919475-10bb-4d20-a936-30e2a871b69f"
Nov 02 18:45:18.892547 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.892525    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq" event={"ID":"55e24920-a5e4-46a2-b782-3eb5d22f3b1e","Type":"ContainerDied","Data":"465fbead6128008355176a2462e64834e566f7d501501c08d9a87759d0c58b76"}
Nov 02 18:45:18.892652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.892566    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"
Nov 02 18:45:18.899240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.899223    4476 scope.go:122] "RemoveContainer" containerID="d4689ff9a694040033f8ab51edae341e2d51796c750c6b9e83970e8ccdcd3074"
Nov 02 18:45:18.910752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.910733    4476 scope.go:122] "RemoveContainer" containerID="9f26e140fb1c1cfde606201f3ebcf7166be7aef72967f929aeb063ef69b05208"
Nov 02 18:45:18.921462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.921443    4476 scope.go:122] "RemoveContainer" containerID="b784c44871fe29ba54575f8955d6d9f693114f2681c07e73cb2d48c855ffbfaa"
Nov 02 18:45:18.933005 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.932981    4476 scope.go:122] "RemoveContainer" containerID="2ee5b443786d68b45104929f200fc50235465737798fc0446d966e295bff3269"
Nov 02 18:45:18.958603 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:18.958567    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"]
Nov 02 18:45:19.015672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.015647    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"]
Nov 02 18:45:19.019225 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.019204    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-413a4e-29894085-wl9vq"]
Nov 02 18:45:19.070508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.070475    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"]
Nov 02 18:45:19.358728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.358691    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"]
Nov 02 18:45:19.359806 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.359787    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894085-bncvm"]
Nov 02 18:45:19.387523 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.387483    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" path="/var/lib/kubelet/pods/49d58be1-880c-4fc2-8104-ddc4d02a3adf/volumes"
Nov 02 18:45:19.388132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.388108    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" path="/var/lib/kubelet/pods/55e24920-a5e4-46a2-b782-3eb5d22f3b1e/volumes"
Nov 02 18:45:19.582303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.582271    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"]
Nov 02 18:45:19.587505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.587476    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894085-76vc6"]
Nov 02 18:45:19.999779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:19.999738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:20.139882 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.139859    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:20.143411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.143389    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:20.256145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.255248    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"]
Nov 02 18:45:20.278119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.278078    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca\") pod \"80674c25-365d-4314-8b60-8e393f630983\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") "
Nov 02 18:45:20.278314 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.278151    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca\") pod \"71919475-10bb-4d20-a936-30e2a871b69f\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") "
Nov 02 18:45:20.278403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.278351    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt\" (UniqueName: \"kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt\") pod \"80674c25-365d-4314-8b60-8e393f630983\" (UID: \"80674c25-365d-4314-8b60-8e393f630983\") "
Nov 02 18:45:20.278468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.278427    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6\" (UniqueName: \"kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6\") pod \"71919475-10bb-4d20-a936-30e2a871b69f\" (UID: \"71919475-10bb-4d20-a936-30e2a871b69f\") "
Nov 02 18:45:20.279667 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.279603    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca" pod "71919475-10bb-4d20-a936-30e2a871b69f" (UID: "71919475-10bb-4d20-a936-30e2a871b69f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:20.279954 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.279907    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca" pod "80674c25-365d-4314-8b60-8e393f630983" (UID: "80674c25-365d-4314-8b60-8e393f630983"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:20.286691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.282397    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt" pod "80674c25-365d-4314-8b60-8e393f630983" (UID: "80674c25-365d-4314-8b60-8e393f630983"). InnerVolumeSpecName "kube-api-access-2htzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:20.291318 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.291285    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6" pod "71919475-10bb-4d20-a936-30e2a871b69f" (UID: "71919475-10bb-4d20-a936-30e2a871b69f"). InnerVolumeSpecName "kube-api-access-ggcp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:20.380650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.380587    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71919475-10bb-4d20-a936-30e2a871b69f-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.380650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.380620    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htzt\" (UniqueName: \"kubernetes.io/projected/80674c25-365d-4314-8b60-8e393f630983-kube-api-access-2htzt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.380650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.380655    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggcp6\" (UniqueName: \"kubernetes.io/projected/71919475-10bb-4d20-a936-30e2a871b69f-kube-api-access-ggcp6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.380963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.380669    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80674c25-365d-4314-8b60-8e393f630983-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.421073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.421048    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:20.481694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.481663    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6\" (UniqueName: \"kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6\") pod \"2e372001-8a06-47f1-ba3b-17ee0f102102\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") "
Nov 02 18:45:20.481877 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.481736    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca\") pod \"2e372001-8a06-47f1-ba3b-17ee0f102102\" (UID: \"2e372001-8a06-47f1-ba3b-17ee0f102102\") "
Nov 02 18:45:20.482723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.482695    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca" pod "2e372001-8a06-47f1-ba3b-17ee0f102102" (UID: "2e372001-8a06-47f1-ba3b-17ee0f102102"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:20.484851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.484819    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6" pod "2e372001-8a06-47f1-ba3b-17ee0f102102" (UID: "2e372001-8a06-47f1-ba3b-17ee0f102102"). InnerVolumeSpecName "kube-api-access-j4dg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:20.583465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.583394    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4dg6\" (UniqueName: \"kubernetes.io/projected/2e372001-8a06-47f1-ba3b-17ee0f102102-kube-api-access-j4dg6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.583465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.583428    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e372001-8a06-47f1-ba3b-17ee0f102102-maas-api-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:20.916510 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.916475    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"
Nov 02 18:45:20.916764 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.916510    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv" event={"ID":"71919475-10bb-4d20-a936-30e2a871b69f","Type":"ContainerDied","Data":"f9521716b5046d44f8c43aebbbfa6212b4acf50e8acf28aea0a52027cbadfc34"}
Nov 02 18:45:20.916764 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.916575    4476 scope.go:122] "RemoveContainer" containerID="97452782cadf912113dbedb66a6c7433658277d28f4134b20d719dd3768f76d7"
Nov 02 18:45:20.918250 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.918232    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"
Nov 02 18:45:20.918352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.918286    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh" event={"ID":"80674c25-365d-4314-8b60-8e393f630983","Type":"ContainerDied","Data":"c6a1bb7b55e9fa186360af57eea0825a64053042beda5e1abc02860724398800"}
Nov 02 18:45:20.919841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.919817    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt" event={"ID":"2e372001-8a06-47f1-ba3b-17ee0f102102","Type":"ContainerDied","Data":"6acc1983a42585faf1dfcd1dbbba53f02d14c5029f662ead2b6f264eab4e3c27"}
Nov 02 18:45:20.919968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.919852    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"
Nov 02 18:45:20.928745 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.928727    4476 scope.go:122] "RemoveContainer" containerID="52f9311f4fbf486d5c34f26791f297043fbec474944162a9bfcd946cce5eb16c"
Nov 02 18:45:20.938541 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:20.938524    4476 scope.go:122] "RemoveContainer" containerID="6a2db4b4f3243960a2347a7bdc4cca4f91dff896556e511b5d4f52be0dfeda29"
Nov 02 18:45:21.080337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.080291    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"]
Nov 02 18:45:21.086284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.086255    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-134555-29894085-q55zh"]
Nov 02 18:45:21.193747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.193619    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"]
Nov 02 18:45:21.208103 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.208065    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894085-cnscv"]
Nov 02 18:45:21.312349 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.312317    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"]
Nov 02 18:45:21.315713 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.315684    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894085-rxzbt"]
Nov 02 18:45:21.386719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.386684    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" path="/var/lib/kubelet/pods/03e3299b-30b4-43da-a27c-60c85b034f4e/volumes"
Nov 02 18:45:21.387164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.387150    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" path="/var/lib/kubelet/pods/2e372001-8a06-47f1-ba3b-17ee0f102102/volumes"
Nov 02 18:45:21.387534 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.387520    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71919475-10bb-4d20-a936-30e2a871b69f" path="/var/lib/kubelet/pods/71919475-10bb-4d20-a936-30e2a871b69f/volumes"
Nov 02 18:45:21.388263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:21.388247    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80674c25-365d-4314-8b60-8e393f630983" path="/var/lib/kubelet/pods/80674c25-365d-4314-8b60-8e393f630983/volumes"
Nov 02 18:45:24.696931 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:24.696896    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:29.061271 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:29.061230    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:45:29.364885 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:29.364844    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:34.038459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:34.038398    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:34.049783 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:34.049753    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"]
Nov 02 18:45:34.408234 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:34.408203    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:38.745328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:38.745297    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:43.414052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:43.414019    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:48.103400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:48.103364    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:49.070776 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.070721    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn" podUID="dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" containerName="main" containerID="cri-o://de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92" gracePeriod=45
Nov 02 18:45:49.337404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.337380    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:45:49.435336 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435308    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9\" (UniqueName: \"kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9\") pod \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") "
Nov 02 18:45:49.435508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435347    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs\") pod \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") "
Nov 02 18:45:49.435508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435434    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache\") pod \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") "
Nov 02 18:45:49.435508 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435490    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir\") pod \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") "
Nov 02 18:45:49.435733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435525    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm\") pod \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") "
Nov 02 18:45:49.435733 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435685    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home\") pod \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\" (UID: \"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3\") "
Nov 02 18:45:49.435861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435758    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir" pod "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" (UID: "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:45:49.435861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435769    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache" pod "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" (UID: "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:45:49.436025 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.435998    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home" pod "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" (UID: "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:45:49.436356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.436334    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:49.436438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.436362    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:49.436438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.436378    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:49.437894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.437867    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm" pod "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" (UID: "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:45:49.437894 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.437883    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs" pod "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" (UID: "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:45:49.438052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.437958    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9" pod "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" (UID: "dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3"). InnerVolumeSpecName "kube-api-access-xstk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:49.538123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.538083    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstk9\" (UniqueName: \"kubernetes.io/projected/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-kube-api-access-xstk9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:49.538123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.538112    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:49.538123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:49.538128    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:45:50.167393 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.167359    4476 generic.go:356] "Generic (PLEG): container finished" podID="dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" containerID="de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92" exitCode=0
Nov 02 18:45:50.167616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.167407    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"
Nov 02 18:45:50.167616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.167420    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn" event={"ID":"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3","Type":"ContainerDied","Data":"de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92"}
Nov 02 18:45:50.167616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.167452    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn" event={"ID":"dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3","Type":"ContainerDied","Data":"ca29cd1bfebc655a6359426608455c486157c98896a96febe8854e45692d4d1b"}
Nov 02 18:45:50.167616 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.167467    4476 scope.go:122] "RemoveContainer" containerID="de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92"
Nov 02 18:45:50.178268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.178245    4476 scope.go:122] "RemoveContainer" containerID="de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92"
Nov 02 18:45:50.178821 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:50.178794    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92\": container with ID starting with de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92 not found: ID does not exist" containerID="de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92"
Nov 02 18:45:50.178947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.178831    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92"} err="failed to get container status \"de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92\": rpc error: code = NotFound desc = could not find container \"de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92\": container with ID starting with de765d5297561aba7682e9141d038c5993944084b44ed53635d435cbd0472a92 not found: ID does not exist"
Nov 02 18:45:50.334297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.334261    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"]
Nov 02 18:45:50.385719 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:50.385682    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-134555/test-model-134555-kserve-5b94869b9c-nspvn"]
Nov 02 18:45:51.385465 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:51.385431    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" path="/var/lib/kubelet/pods/dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3/volumes"
Nov 02 18:45:52.797166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:52.797128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:57.501191 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:45:57.501157    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:45:59.059533 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:45:59.059495    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:46:02.206144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:02.206069    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:03.796838 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:03.796774    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:06.892151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:06.892117    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:08.209965 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:08.209933    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:08.505053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:08.504972    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:10.059781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:10.059742    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:10.594602 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:10.594568    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:10.929773 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:10.929740    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:11.808814 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:11.808786    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:15.619860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:15.619813    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:16.425165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:16.425131    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:16.735516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:16.735436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:21.253425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:21.253384    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:22.317587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:22.317552    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:22.628653 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:22.628601    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:22.945280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:22.945195    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:26.656675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:26.656616    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:28.994219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:28.994190    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:29.060977 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:46:29.060942    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:46:31.418528 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:31.418488    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:32.430757 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:32.430722    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:36.106068 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:36.106037    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:40.832928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:40.832889    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:45.605347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:45.605315    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:50.308247 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:50.308207    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:53.815912 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.815877    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"]
Nov 02 18:46:53.816449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816433    4476 cpu_manager.go:414] "containerMap: removing container" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816452    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816463    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816468    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816475    4476 cpu_manager.go:414] "containerMap: removing container" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816480    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816495    4476 cpu_manager.go:414] "containerMap: removing container" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:46:53.816499 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816500    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816507    4476 cpu_manager.go:414] "containerMap: removing container" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816512    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816523    4476 cpu_manager.go:414] "containerMap: removing container" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816527    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816538    4476 cpu_manager.go:414] "containerMap: removing container" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816543    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816548    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816553    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816561    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816565    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816574    4476 cpu_manager.go:414] "containerMap: removing container" podUID="dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" containerName="main"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816579    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" containerName="main"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816589    4476 cpu_manager.go:414] "containerMap: removing container" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816593    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816601    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816605    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816613    4476 cpu_manager.go:414] "containerMap: removing container" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816617    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816622    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816643    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816735    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816743    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816748    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816754    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816760    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816765    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816771    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:46:53.816768 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816779    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816785    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816790    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816797    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816803    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816809    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816813    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816818    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e3299b-30b4-43da-a27c-60c85b034f4e" containerName="cleanup"
Nov 02 18:46:53.817671 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.816824    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbdba31a-15fa-4f08-af6b-cfb3d3c0bdb3" containerName="main"
Nov 02 18:46:53.820328 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.820312    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.839022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.836480    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.839022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.836691    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ql5\" (UniqueName: \"kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.839022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.836772    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.875753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.875725    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"]
Nov 02 18:46:53.919303 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.919276    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:46:53.919454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.919325    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\""
Nov 02 18:46:53.937594 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.937563    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.937762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.937679    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ql5\" (UniqueName: \"kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.937762 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.937731    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.938526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.938499    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.940189 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.940165    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:53.994445 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:53.994417    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ql5\" (UniqueName: \"kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5\") pod \"maas-api-revoke-sub-545aabe6-c4kkb\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:54.133718 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:54.133691    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:54.515360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:54.515336    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"]
Nov 02 18:46:54.516850 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:46:54.516819    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdea7084_f2eb_4883_ae0e_132fd5041b96.slice/crio-379babb39fa8803253a86aeef58e8d130c061f2a410b0151a05ddeafb4072d6b WatchSource:0}: Error finding container 379babb39fa8803253a86aeef58e8d130c061f2a410b0151a05ddeafb4072d6b: Status 404 returned error can't find the container with id 379babb39fa8803253a86aeef58e8d130c061f2a410b0151a05ddeafb4072d6b
Nov 02 18:46:54.695524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:54.695487    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb" event={"ID":"fdea7084-f2eb-4883-ae0e-132fd5041b96","Type":"ContainerStarted","Data":"379babb39fa8803253a86aeef58e8d130c061f2a410b0151a05ddeafb4072d6b"}
Nov 02 18:46:54.949464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:54.949426    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:46:55.703425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:55.703329    4476 generic.go:356] "Generic (PLEG): container finished" podID="fdea7084-f2eb-4883-ae0e-132fd5041b96" containerID="c6dd2413695581fdb83ee9959a5c5d5819697afdef3888a231a9582c98b9cffd" exitCode=0
Nov 02 18:46:55.703600 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:55.703417    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb" event={"ID":"fdea7084-f2eb-4883-ae0e-132fd5041b96","Type":"ContainerDied","Data":"c6dd2413695581fdb83ee9959a5c5d5819697afdef3888a231a9582c98b9cffd"}
Nov 02 18:46:56.895043 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.895014    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:56.975136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.975029    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca\") pod \"fdea7084-f2eb-4883-ae0e-132fd5041b96\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") "
Nov 02 18:46:56.975136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.975089    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca\") pod \"fdea7084-f2eb-4883-ae0e-132fd5041b96\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") "
Nov 02 18:46:56.975395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.975232    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5\" (UniqueName: \"kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5\") pod \"fdea7084-f2eb-4883-ae0e-132fd5041b96\" (UID: \"fdea7084-f2eb-4883-ae0e-132fd5041b96\") "
Nov 02 18:46:56.975552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.975519    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca" pod "fdea7084-f2eb-4883-ae0e-132fd5041b96" (UID: "fdea7084-f2eb-4883-ae0e-132fd5041b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:56.975755 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.975736    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:56.977505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.977478    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5" pod "fdea7084-f2eb-4883-ae0e-132fd5041b96" (UID: "fdea7084-f2eb-4883-ae0e-132fd5041b96"). InnerVolumeSpecName "kube-api-access-r5ql5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:56.977835 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:56.977809    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca" pod "fdea7084-f2eb-4883-ae0e-132fd5041b96" (UID: "fdea7084-f2eb-4883-ae0e-132fd5041b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:46:57.077079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:57.077041    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ql5\" (UniqueName: \"kubernetes.io/projected/fdea7084-f2eb-4883-ae0e-132fd5041b96-kube-api-access-r5ql5\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:57.077079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:57.077078    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdea7084-f2eb-4883-ae0e-132fd5041b96-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:46:57.718844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:57.718770    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"
Nov 02 18:46:57.718844 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:57.718809    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb" event={"ID":"fdea7084-f2eb-4883-ae0e-132fd5041b96","Type":"ContainerDied","Data":"379babb39fa8803253a86aeef58e8d130c061f2a410b0151a05ddeafb4072d6b"}
Nov 02 18:46:57.719300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:57.718860    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="379babb39fa8803253a86aeef58e8d130c061f2a410b0151a05ddeafb4072d6b"
Nov 02 18:46:59.061214 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:46:59.061163    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:46:59.692473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:46:59.692438    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:03.776350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:03.776314    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:05.521140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:05.521105    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:05.942810 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:05.942779    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:06.264864 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:06.264762    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:07.497612 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:07.497571    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:08.101410 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:08.101371    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:08.557281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:08.557249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:08.934589 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:08.934550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:09.897435 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:09.897400    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:10.199073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:10.198992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:10.697288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:10.697255    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:15.077202 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:15.077166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:15.508670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:15.508613    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:16.315982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:16.315951    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:16.624705 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:16.624677    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:20.473850 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:20.473815    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:20.798647 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:20.798531    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:25.109028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:25.108988    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:28.448385 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:28.448349    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:28.774739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:28.774602    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:29.062708 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:47:29.062593    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:47:29.266016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:29.265978    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:31.016412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:31.016128    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:35.704266 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:35.704230    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:36.198947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:36.198909    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:36.544227 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:36.544136    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:37.711449 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:37.711420    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:38.175473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:38.175439    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:38.928959 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:38.928927    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:39.987584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:39.987546    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"]
Nov 02 18:47:39.997901 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:39.997863    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:40.743800 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:40.741115    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:40.928071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:40.928041    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"]
Nov 02 18:47:41.137230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:41.137194    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:44.940371 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:44.940331    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:45.735156 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:45.735122    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:46.434886 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:46.434851    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"]
Nov 02 18:47:46.545915 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:46.545882    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"]
Nov 02 18:47:46.793109 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:46.793025    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:47.098779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:47.098684    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:50.273144 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:50.273108    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:47:50.282516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:50.282489    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:47:50.406970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:50.406935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:50.711319 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:50.711286    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:51.438181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.438108    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8" podUID="4b823f43-2b94-4f45-a604-fed01945ae56" containerName="payload-processing" containerID="cri-o://80dd1d0bf2bc92c57336b8aa7057f2395e0f6e6cc7731fe56715bd89b40f051d" gracePeriod=25
Nov 02 18:47:51.547195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.547140    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerName="payload-pre-processing" containerID="cri-o://0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48" gracePeriod=25
Nov 02 18:47:51.708658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.708614    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:47:51.737675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.737645    4476 scope.go:122] "RemoveContainer" containerID="80dd1d0bf2bc92c57336b8aa7057f2395e0f6e6cc7731fe56715bd89b40f051d"
Nov 02 18:47:51.789577 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.789534    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k\" (UniqueName: \"kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k\") pod \"4b823f43-2b94-4f45-a604-fed01945ae56\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") "
Nov 02 18:47:51.789786 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.789610    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume\") pod \"4b823f43-2b94-4f45-a604-fed01945ae56\" (UID: \"4b823f43-2b94-4f45-a604-fed01945ae56\") "
Nov 02 18:47:51.790052 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.790023    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume" pod "4b823f43-2b94-4f45-a604-fed01945ae56" (UID: "4b823f43-2b94-4f45-a604-fed01945ae56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:47:51.791945 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.791913    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k" pod "4b823f43-2b94-4f45-a604-fed01945ae56" (UID: "4b823f43-2b94-4f45-a604-fed01945ae56"). InnerVolumeSpecName "kube-api-access-xvj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:51.891081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.891053    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvj8k\" (UniqueName: \"kubernetes.io/projected/4b823f43-2b94-4f45-a604-fed01945ae56-kube-api-access-xvj8k\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:51.891081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:51.891078    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b823f43-2b94-4f45-a604-fed01945ae56-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:52.205083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:52.205037    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8" event={"ID":"4b823f43-2b94-4f45-a604-fed01945ae56","Type":"ContainerDied","Data":"80dd1d0bf2bc92c57336b8aa7057f2395e0f6e6cc7731fe56715bd89b40f051d"}
Nov 02 18:47:52.205267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:52.205111    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"
Nov 02 18:47:52.205267 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:52.205126    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8" event={"ID":"4b823f43-2b94-4f45-a604-fed01945ae56","Type":"ContainerDied","Data":"c9ea747e7bd13e3e2e1c290495195783fad38f23b5b2870f5a3e82f808636a62"}
Nov 02 18:47:52.431058 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:52.431026    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"]
Nov 02 18:47:52.435546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:52.435523    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fe24d-759b674f-hcdf8"]
Nov 02 18:47:53.385927 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:53.385888    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b823f43-2b94-4f45-a604-fed01945ae56" path="/var/lib/kubelet/pods/4b823f43-2b94-4f45-a604-fed01945ae56/volumes"
Nov 02 18:47:55.015858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.015824    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6" podUID="d2f06102-0149-4901-aaa5-e13243f51838" containerName="main" containerID="cri-o://4ba21da7d79df2a003386a5515d893d2116ebd7483455189dfe515e018d27092" gracePeriod=45
Nov 02 18:47:55.229348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.229308    4476 generic.go:356] "Generic (PLEG): container finished" podID="d2f06102-0149-4901-aaa5-e13243f51838" containerID="4ba21da7d79df2a003386a5515d893d2116ebd7483455189dfe515e018d27092" exitCode=0
Nov 02 18:47:55.229522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.229389    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6" event={"ID":"d2f06102-0149-4901-aaa5-e13243f51838","Type":"ContainerDied","Data":"4ba21da7d79df2a003386a5515d893d2116ebd7483455189dfe515e018d27092"}
Nov 02 18:47:55.243704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.243656    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:55.288147 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.288124    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:47:55.331852 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.331824    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache\") pod \"d2f06102-0149-4901-aaa5-e13243f51838\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") "
Nov 02 18:47:55.332063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.331905    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb\" (UniqueName: \"kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb\") pod \"d2f06102-0149-4901-aaa5-e13243f51838\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") "
Nov 02 18:47:55.332063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332013    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs\") pod \"d2f06102-0149-4901-aaa5-e13243f51838\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") "
Nov 02 18:47:55.332063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332052    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm\") pod \"d2f06102-0149-4901-aaa5-e13243f51838\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") "
Nov 02 18:47:55.332258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332108    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache" pod "d2f06102-0149-4901-aaa5-e13243f51838" (UID: "d2f06102-0149-4901-aaa5-e13243f51838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:55.332258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332112    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir\") pod \"d2f06102-0149-4901-aaa5-e13243f51838\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") "
Nov 02 18:47:55.332258 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332166    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home\") pod \"d2f06102-0149-4901-aaa5-e13243f51838\" (UID: \"d2f06102-0149-4901-aaa5-e13243f51838\") "
Nov 02 18:47:55.332500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332341    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir" pod "d2f06102-0149-4901-aaa5-e13243f51838" (UID: "d2f06102-0149-4901-aaa5-e13243f51838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:55.332500 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332412    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home" pod "d2f06102-0149-4901-aaa5-e13243f51838" (UID: "d2f06102-0149-4901-aaa5-e13243f51838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:55.332777 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332755    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.332902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332779    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.332902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.332793    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.334413 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.334390    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm" pod "d2f06102-0149-4901-aaa5-e13243f51838" (UID: "d2f06102-0149-4901-aaa5-e13243f51838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:55.334509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.334393    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb" pod "d2f06102-0149-4901-aaa5-e13243f51838" (UID: "d2f06102-0149-4901-aaa5-e13243f51838"). InnerVolumeSpecName "kube-api-access-fldjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:55.334509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.334444    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs" pod "d2f06102-0149-4901-aaa5-e13243f51838" (UID: "d2f06102-0149-4901-aaa5-e13243f51838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:55.434151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.434108    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fldjb\" (UniqueName: \"kubernetes.io/projected/d2f06102-0149-4901-aaa5-e13243f51838-kube-api-access-fldjb\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.434151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.434143    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f06102-0149-4901-aaa5-e13243f51838-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.434151 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.434157    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f06102-0149-4901-aaa5-e13243f51838-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.970423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:55.970356    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" podUID="fab4a298-dba5-4162-b226-e89896848f14" containerName="main" containerID="cri-o://0e0cc5a590ff704823e4e308176100f489c24a5e7feb9717b1f696f82815875b" gracePeriod=45
Nov 02 18:47:56.108374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.108328    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" podUID="fab4a298-dba5-4162-b226-e89896848f14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.234:8000/ready\": dial tcp 10.133.0.234:8000: connect: connection refused"
Nov 02 18:47:56.240876 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.240842    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"
Nov 02 18:47:56.241048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.240852    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6" event={"ID":"d2f06102-0149-4901-aaa5-e13243f51838","Type":"ContainerDied","Data":"3dd3742cda7d849518e0e07d378c1c9a07828a097b4ab0ffeaca6d7d898359e6"}
Nov 02 18:47:56.241048 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.240984    4476 scope.go:122] "RemoveContainer" containerID="4ba21da7d79df2a003386a5515d893d2116ebd7483455189dfe515e018d27092"
Nov 02 18:47:56.242933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.242896    4476 generic.go:356] "Generic (PLEG): container finished" podID="fab4a298-dba5-4162-b226-e89896848f14" containerID="0e0cc5a590ff704823e4e308176100f489c24a5e7feb9717b1f696f82815875b" exitCode=0
Nov 02 18:47:56.243064 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.242992    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" event={"ID":"fab4a298-dba5-4162-b226-e89896848f14","Type":"ContainerDied","Data":"0e0cc5a590ff704823e4e308176100f489c24a5e7feb9717b1f696f82815875b"}
Nov 02 18:47:56.266167 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.266142    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:47:56.343807 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.343765    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home\") pod \"fab4a298-dba5-4162-b226-e89896848f14\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") "
Nov 02 18:47:56.344001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.343854    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache\") pod \"fab4a298-dba5-4162-b226-e89896848f14\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") "
Nov 02 18:47:56.344001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.343893    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs\") pod \"fab4a298-dba5-4162-b226-e89896848f14\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") "
Nov 02 18:47:56.344091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344068    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8\" (UniqueName: \"kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8\") pod \"fab4a298-dba5-4162-b226-e89896848f14\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") "
Nov 02 18:47:56.344128 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home" pod "fab4a298-dba5-4162-b226-e89896848f14" (UID: "fab4a298-dba5-4162-b226-e89896848f14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:56.344170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344142    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir\") pod \"fab4a298-dba5-4162-b226-e89896848f14\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") "
Nov 02 18:47:56.344170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344141    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache" pod "fab4a298-dba5-4162-b226-e89896848f14" (UID: "fab4a298-dba5-4162-b226-e89896848f14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:56.344249 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344170    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm\") pod \"fab4a298-dba5-4162-b226-e89896848f14\" (UID: \"fab4a298-dba5-4162-b226-e89896848f14\") "
Nov 02 18:47:56.344374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344357    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir" pod "fab4a298-dba5-4162-b226-e89896848f14" (UID: "fab4a298-dba5-4162-b226-e89896848f14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:56.344666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344608    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:56.344666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344664    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:56.345014 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.344680    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:56.346438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.346404    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs" pod "fab4a298-dba5-4162-b226-e89896848f14" (UID: "fab4a298-dba5-4162-b226-e89896848f14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:56.346552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.346453    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8" pod "fab4a298-dba5-4162-b226-e89896848f14" (UID: "fab4a298-dba5-4162-b226-e89896848f14"). InnerVolumeSpecName "kube-api-access-hxhs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:56.346552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.346481    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm" pod "fab4a298-dba5-4162-b226-e89896848f14" (UID: "fab4a298-dba5-4162-b226-e89896848f14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:47:56.404609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.404563    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"]
Nov 02 18:47:56.446079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.446038    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab4a298-dba5-4162-b226-e89896848f14-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:56.446079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.446071    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab4a298-dba5-4162-b226-e89896848f14-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:56.446297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.446085    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhs8\" (UniqueName: \"kubernetes.io/projected/fab4a298-dba5-4162-b226-e89896848f14-kube-api-access-hxhs8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:47:56.453809 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:56.453774    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-413a4e/sub-test-model-413a4e-kserve-646b84db85-q6ts6"]
Nov 02 18:47:57.254329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:57.254293    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd" event={"ID":"fab4a298-dba5-4162-b226-e89896848f14","Type":"ContainerDied","Data":"cf8db434cce66d41e8f376b3877e51db32a9d20e65117d3d1759206cd8e2e40e"}
Nov 02 18:47:57.254329 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:57.254332    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"
Nov 02 18:47:57.254933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:57.254357    4476 scope.go:122] "RemoveContainer" containerID="0e0cc5a590ff704823e4e308176100f489c24a5e7feb9717b1f696f82815875b"
Nov 02 18:47:57.383129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:57.383091    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f06102-0149-4901-aaa5-e13243f51838" path="/var/lib/kubelet/pods/d2f06102-0149-4901-aaa5-e13243f51838/volumes"
Nov 02 18:47:57.417236 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:57.417200    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"]
Nov 02 18:47:57.466586 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:57.466544    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-580a21/sub-test-model-580a21-kserve-654f89b8c-q5pwd"]
Nov 02 18:47:59.060010 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:47:59.059970    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:47:59.383966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:59.383936    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab4a298-dba5-4162-b226-e89896848f14" path="/var/lib/kubelet/pods/fab4a298-dba5-4162-b226-e89896848f14/volumes"
Nov 02 18:47:59.575185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:59.575137    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:59.894490 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:59.894461    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:47:59.997675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:47:59.997607    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:48:00.198922 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:00.198850    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:04.605834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:04.605769    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:08.913357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:08.913299    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:09.315654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:09.315547    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:09.998054 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:09.998018    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:48:10.272990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:10.272895    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"]
Nov 02 18:48:10.286888 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:10.286854    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:10.722259 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:10.722222    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:11.802607 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:11.802543    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:48:11.929079 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:11.929048    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq\" (UniqueName: \"kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq\") pod \"b80b6429-fd80-4c59-a1a6-b0b991667f03\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") "
Nov 02 18:48:11.929293 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:11.929153    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume\") pod \"b80b6429-fd80-4c59-a1a6-b0b991667f03\" (UID: \"b80b6429-fd80-4c59-a1a6-b0b991667f03\") "
Nov 02 18:48:11.929622 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:11.929590    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume" pod "b80b6429-fd80-4c59-a1a6-b0b991667f03" (UID: "b80b6429-fd80-4c59-a1a6-b0b991667f03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:11.931433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:11.931404    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq" pod "b80b6429-fd80-4c59-a1a6-b0b991667f03" (UID: "b80b6429-fd80-4c59-a1a6-b0b991667f03"). InnerVolumeSpecName "kube-api-access-28qlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:11.940772 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:11.940746    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:12.030351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.030312    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28qlq\" (UniqueName: \"kubernetes.io/projected/b80b6429-fd80-4c59-a1a6-b0b991667f03-kube-api-access-28qlq\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:12.030351 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.030345    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80b6429-fd80-4c59-a1a6-b0b991667f03-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:12.368200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.368169    4476 generic.go:356] "Generic (PLEG): container finished" podID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerID="0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48" exitCode=0
Nov 02 18:48:12.368392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.368212    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"
Nov 02 18:48:12.368392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.368246    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" event={"ID":"b80b6429-fd80-4c59-a1a6-b0b991667f03","Type":"ContainerDied","Data":"0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48"}
Nov 02 18:48:12.368392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.368321    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln" event={"ID":"b80b6429-fd80-4c59-a1a6-b0b991667f03","Type":"ContainerDied","Data":"c8abcb4d518eef3271bf9f80352285a6d2fed8711cbfc3e5b08a5f362abf7804"}
Nov 02 18:48:12.368392 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.368347    4476 scope.go:122] "RemoveContainer" containerID="0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48"
Nov 02 18:48:12.379348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.379325    4476 scope.go:122] "RemoveContainer" containerID="0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48"
Nov 02 18:48:12.379671 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:48:12.379645    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48\": container with ID starting with 0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48 not found: ID does not exist" containerID="0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48"
Nov 02 18:48:12.379754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.379685    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48"} err="failed to get container status \"0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48\": rpc error: code = NotFound desc = could not find container \"0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48\": container with ID starting with 0d6819ab0f422ff8cd19e2407b4c4f1c25eea2f71de3699bdaa16e6c38f5fa48 not found: ID does not exist"
Nov 02 18:48:12.533515 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.533480    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"]
Nov 02 18:48:12.585205 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:12.585166    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fe24d-844fc67fbc-swbln"]
Nov 02 18:48:13.383365 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:13.383323    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" path="/var/lib/kubelet/pods/b80b6429-fd80-4c59-a1a6-b0b991667f03/volumes"
Nov 02 18:48:14.076896 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:14.076859    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:16.789437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:16.789402    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:17.625095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:17.625062    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:18.235742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:18.235713    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:18.553444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:18.553359    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:19.107622 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:19.107586    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:19.779990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:19.779957    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"]
Nov 02 18:48:20.029968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:20.029934    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:20.357268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:20.357229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:20.785055 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:20.785022    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:22.594595 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:22.594562    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:23.462582 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:23.462550    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:24.411093 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:24.411054    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:24.781137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:24.781042    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" podUID="001c983b-7313-493d-9fec-04b983993105" containerName="payload-pre-processing" containerID="cri-o://c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4" gracePeriod=25
Nov 02 18:48:25.209081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.209037    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:25.296841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.296805    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv" podUID="642a337b-9bbb-4202-bc14-57bfa7869568" containerName="main" containerID="cri-o://9b1990ba0b7dadc1fd53eed3e5cc74abd48f6d05d8e8a323e571da969c91b76b" gracePeriod=45
Nov 02 18:48:25.480836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.480791    4476 generic.go:356] "Generic (PLEG): container finished" podID="642a337b-9bbb-4202-bc14-57bfa7869568" containerID="9b1990ba0b7dadc1fd53eed3e5cc74abd48f6d05d8e8a323e571da969c91b76b" exitCode=0
Nov 02 18:48:25.481287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.480986    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv" event={"ID":"642a337b-9bbb-4202-bc14-57bfa7869568","Type":"ContainerDied","Data":"9b1990ba0b7dadc1fd53eed3e5cc74abd48f6d05d8e8a323e571da969c91b76b"}
Nov 02 18:48:25.572017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.571988    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:48:25.605072 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605042    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs\" (UniqueName: \"kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs\") pod \"642a337b-9bbb-4202-bc14-57bfa7869568\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") "
Nov 02 18:48:25.605221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605088    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home\") pod \"642a337b-9bbb-4202-bc14-57bfa7869568\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") "
Nov 02 18:48:25.605221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605188    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm\") pod \"642a337b-9bbb-4202-bc14-57bfa7869568\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") "
Nov 02 18:48:25.605221 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605220    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir\") pod \"642a337b-9bbb-4202-bc14-57bfa7869568\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") "
Nov 02 18:48:25.605387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605255    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk\" (UniqueName: \"kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk\") pod \"642a337b-9bbb-4202-bc14-57bfa7869568\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") "
Nov 02 18:48:25.605387 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605345    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache\") pod \"642a337b-9bbb-4202-bc14-57bfa7869568\" (UID: \"642a337b-9bbb-4202-bc14-57bfa7869568\") "
Nov 02 18:48:25.605504 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605391    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home" pod "642a337b-9bbb-4202-bc14-57bfa7869568" (UID: "642a337b-9bbb-4202-bc14-57bfa7869568"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:25.605742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605720    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache" pod "642a337b-9bbb-4202-bc14-57bfa7869568" (UID: "642a337b-9bbb-4202-bc14-57bfa7869568"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:25.605821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605733    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir" pod "642a337b-9bbb-4202-bc14-57bfa7869568" (UID: "642a337b-9bbb-4202-bc14-57bfa7869568"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:25.605879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605834    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:25.605879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605855    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:25.605879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.605868    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:25.607398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.607374    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm" pod "642a337b-9bbb-4202-bc14-57bfa7869568" (UID: "642a337b-9bbb-4202-bc14-57bfa7869568"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:25.607520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.607403    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs" pod "642a337b-9bbb-4202-bc14-57bfa7869568" (UID: "642a337b-9bbb-4202-bc14-57bfa7869568"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:25.607704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.607687    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk" pod "642a337b-9bbb-4202-bc14-57bfa7869568" (UID: "642a337b-9bbb-4202-bc14-57bfa7869568"). InnerVolumeSpecName "kube-api-access-4ljmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:25.707520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.707482    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/642a337b-9bbb-4202-bc14-57bfa7869568-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:25.707722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.707526    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljmk\" (UniqueName: \"kubernetes.io/projected/642a337b-9bbb-4202-bc14-57bfa7869568-kube-api-access-4ljmk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:25.707722 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:25.707545    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642a337b-9bbb-4202-bc14-57bfa7869568-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:26.491056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:26.491027    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"
Nov 02 18:48:26.491534 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:26.491064    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv" event={"ID":"642a337b-9bbb-4202-bc14-57bfa7869568","Type":"ContainerDied","Data":"44392f931c129ce243fd7db60a50634f80ffae3f2b7cedd673a3ebb82a91fd2b"}
Nov 02 18:48:26.491534 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:26.491116    4476 scope.go:122] "RemoveContainer" containerID="9b1990ba0b7dadc1fd53eed3e5cc74abd48f6d05d8e8a323e571da969c91b76b"
Nov 02 18:48:26.657426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:26.657386    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"]
Nov 02 18:48:26.704462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:26.704428    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c82472/sub-test-model-c82472-kserve-65d97f4bbf-k7dqv"]
Nov 02 18:48:27.385945 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:27.385901    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642a337b-9bbb-4202-bc14-57bfa7869568" path="/var/lib/kubelet/pods/642a337b-9bbb-4202-bc14-57bfa7869568/volumes"
Nov 02 18:48:27.575507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:27.575473    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:27.899287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:27.899257    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:29.059766 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:48:29.059729    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:48:29.160471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:29.160436    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:29.704874 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:29.704813    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:30.141851 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:30.141814    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" podUID="001c983b-7313-493d-9fec-04b983993105" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:48:30.160585 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:30.160553    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:30.515688 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:30.515589    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:35.295678 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:35.295585    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:37.073254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:37.073204    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:37.401730 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:37.401697    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:38.151540 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:38.151507    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:38.466311 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:38.466229    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:40.140919 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:40.140877    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" podUID="001c983b-7313-493d-9fec-04b983993105" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:48:40.303239 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:40.303202    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:42.827335 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:42.827301    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:44.738142 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:44.738108    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"]
Nov 02 18:48:44.738595 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:44.738352    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" podUID="0b26c2bf-2217-4f94-9005-b739811a8667" containerName="maas-api" containerID="cri-o://c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7" gracePeriod=30
Nov 02 18:48:44.825581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:44.825546    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:45.112531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.110171    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:48:45.194358 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.194335    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:48:45.254011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.249828    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca\") pod \"0b26c2bf-2217-4f94-9005-b739811a8667\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") "
Nov 02 18:48:45.254011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.249926    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88\" (UniqueName: \"kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88\") pod \"0b26c2bf-2217-4f94-9005-b739811a8667\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") "
Nov 02 18:48:45.254011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.249983    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\") pod \"0b26c2bf-2217-4f94-9005-b739811a8667\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") "
Nov 02 18:48:45.254011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.250079    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs\") pod \"0b26c2bf-2217-4f94-9005-b739811a8667\" (UID: \"0b26c2bf-2217-4f94-9005-b739811a8667\") "
Nov 02 18:48:45.254011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.253069    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca" pod "0b26c2bf-2217-4f94-9005-b739811a8667" (UID: "0b26c2bf-2217-4f94-9005-b739811a8667"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:45.254355 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.254069    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88" pod "0b26c2bf-2217-4f94-9005-b739811a8667" (UID: "0b26c2bf-2217-4f94-9005-b739811a8667"). InnerVolumeSpecName "kube-api-access-52q88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:45.259900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.255801    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls" pod "0b26c2bf-2217-4f94-9005-b739811a8667" (UID: "0b26c2bf-2217-4f94-9005-b739811a8667"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:45.259900 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.257101    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs" pod "0b26c2bf-2217-4f94-9005-b739811a8667" (UID: "0b26c2bf-2217-4f94-9005-b739811a8667"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:45.351195 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.351097    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume\") pod \"001c983b-7313-493d-9fec-04b983993105\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") "
Nov 02 18:48:45.351542 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.351509    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z\" (UniqueName: \"kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z\") pod \"001c983b-7313-493d-9fec-04b983993105\" (UID: \"001c983b-7313-493d-9fec-04b983993105\") "
Nov 02 18:48:45.352354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.352215    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume" pod "001c983b-7313-493d-9fec-04b983993105" (UID: "001c983b-7313-493d-9fec-04b983993105"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:45.353018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.352995    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52q88\" (UniqueName: \"kubernetes.io/projected/0b26c2bf-2217-4f94-9005-b739811a8667-kube-api-access-52q88\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.353169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.353153    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.353278 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.353265    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b26c2bf-2217-4f94-9005-b739811a8667-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.353383 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.353371    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001c983b-7313-493d-9fec-04b983993105-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.353471 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.353459    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b26c2bf-2217-4f94-9005-b739811a8667-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.354711 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.354603    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z" pod "001c983b-7313-493d-9fec-04b983993105" (UID: "001c983b-7313-493d-9fec-04b983993105"). InnerVolumeSpecName "kube-api-access-f6l5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:45.455230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.455195    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6l5z\" (UniqueName: \"kubernetes.io/projected/001c983b-7313-493d-9fec-04b983993105-kube-api-access-f6l5z\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:45.630666 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.630608    4476 generic.go:356] "Generic (PLEG): container finished" podID="001c983b-7313-493d-9fec-04b983993105" containerID="c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4" exitCode=0
Nov 02 18:48:45.630873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.630689    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"
Nov 02 18:48:45.630873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.630698    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" event={"ID":"001c983b-7313-493d-9fec-04b983993105","Type":"ContainerDied","Data":"c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4"}
Nov 02 18:48:45.630873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.630767    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp" event={"ID":"001c983b-7313-493d-9fec-04b983993105","Type":"ContainerDied","Data":"1da45afa4771c62516452babc9f98e5f8a2dd585ef4db23835d7933e66836b22"}
Nov 02 18:48:45.630873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.630801    4476 scope.go:122] "RemoveContainer" containerID="c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4"
Nov 02 18:48:45.643549 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.643514    4476 generic.go:356] "Generic (PLEG): container finished" podID="0b26c2bf-2217-4f94-9005-b739811a8667" containerID="c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7" exitCode=0
Nov 02 18:48:45.643723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.643573    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"
Nov 02 18:48:45.643723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.643594    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" event={"ID":"0b26c2bf-2217-4f94-9005-b739811a8667","Type":"ContainerDied","Data":"c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7"}
Nov 02 18:48:45.643878 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.643724    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r" event={"ID":"0b26c2bf-2217-4f94-9005-b739811a8667","Type":"ContainerDied","Data":"e6e62866e4cc4ff52b0113b2c25835289653e36b58d5add361c1047317131e59"}
Nov 02 18:48:45.654784 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.654764    4476 scope.go:122] "RemoveContainer" containerID="c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4"
Nov 02 18:48:45.655108 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:48:45.655084    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4\": container with ID starting with c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4 not found: ID does not exist" containerID="c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4"
Nov 02 18:48:45.655168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.655117    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4"} err="failed to get container status \"c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4\": rpc error: code = NotFound desc = could not find container \"c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4\": container with ID starting with c423e7e4e48438b6abf3fa62f6ad2fb80410acffb8122735d8cd32d64e5e00a4 not found: ID does not exist"
Nov 02 18:48:45.655168 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.655134    4476 scope.go:122] "RemoveContainer" containerID="c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7"
Nov 02 18:48:45.667272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.667241    4476 scope.go:122] "RemoveContainer" containerID="c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7"
Nov 02 18:48:45.667574 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:48:45.667547    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7\": container with ID starting with c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7 not found: ID does not exist" containerID="c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7"
Nov 02 18:48:45.667700 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.667575    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7"} err="failed to get container status \"c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7\": rpc error: code = NotFound desc = could not find container \"c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7\": container with ID starting with c440a32e912b0616624834f32f78097d154067f9a1018f94e3b55f498edfd1f7 not found: ID does not exist"
Nov 02 18:48:45.793716 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.793686    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"]
Nov 02 18:48:45.796831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.796800    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-413a4e-666b5776cf-nzjmp"]
Nov 02 18:48:45.903260 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.903189    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"]
Nov 02 18:48:45.907736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:45.907714    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4fe24d-7f5d8bb795-lrq2r"]
Nov 02 18:48:46.013476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:46.013448    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:47.383821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:47.383779    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001c983b-7313-493d-9fec-04b983993105" path="/var/lib/kubelet/pods/001c983b-7313-493d-9fec-04b983993105/volumes"
Nov 02 18:48:47.384376 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:47.384356    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b26c2bf-2217-4f94-9005-b739811a8667" path="/var/lib/kubelet/pods/0b26c2bf-2217-4f94-9005-b739811a8667/volumes"
Nov 02 18:48:47.935061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:47.935028    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:49.341284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:49.341191    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:49.642805 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:49.642775    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:50.605566 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:50.605533    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"]
Nov 02 18:48:50.606015 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:50.605769    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy" containerID="cri-o://9705eaa320085d8fe305beb20f9205e88fe8cd64ed0a9d9c13b667a162cf3af4" gracePeriod=30
Nov 02 18:48:50.828238 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:50.828206    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:51.293983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:51.293942    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:51.827820 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:51.827794    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:52.974513 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:52.974481    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:55.576138 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.576102    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:55.726396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.726227    4476 generic.go:356] "Generic (PLEG): container finished" podID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerID="9705eaa320085d8fe305beb20f9205e88fe8cd64ed0a9d9c13b667a162cf3af4" exitCode=0
Nov 02 18:48:55.726396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.726309    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" event={"ID":"af9e95d7-9f83-4388-b59b-7e14307e88ba","Type":"ContainerDied","Data":"9705eaa320085d8fe305beb20f9205e88fe8cd64ed0a9d9c13b667a162cf3af4"}
Nov 02 18:48:55.877321 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.877298    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:48:55.893968 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.893935    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:55.982218 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982186    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982280    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982395    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982439    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982508    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:55.982554 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982541    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982587    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982619    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982681    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy\") pod \"af9e95d7-9f83-4388-b59b-7e14307e88ba\" (UID: \"af9e95d7-9f83-4388-b59b-7e14307e88ba\") "
Nov 02 18:48:55.982963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.982930    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:55.983073 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.983051    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:55.983324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.983131    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:55.983324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.983153    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:55.983657 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.983596    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:55.984077 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.984056    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:55.985568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.985497    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:48:55.985726 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.985662    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "kube-api-access-znnnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:55.985925 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.985899    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:48:55.986053 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:55.986032    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token" pod "af9e95d7-9f83-4388-b59b-7e14307e88ba" (UID: "af9e95d7-9f83-4388-b59b-7e14307e88ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:56.084551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084518    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnnj\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-kube-api-access-znnnj\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.084551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084547    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e95d7-9f83-4388-b59b-7e14307e88ba-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.084551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084557    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.084822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084571    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.084822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084580    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.084822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084588    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.084822 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.084596    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e95d7-9f83-4388-b59b-7e14307e88ba-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:48:56.734454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.734420    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"
Nov 02 18:48:56.735016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.734455    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9" event={"ID":"af9e95d7-9f83-4388-b59b-7e14307e88ba","Type":"ContainerDied","Data":"1a2fd112f7033b99c9c797c370d5fe7a5f112107c62935c4d017b442fc1d3573"}
Nov 02 18:48:56.735016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.734516    4476 scope.go:122] "RemoveContainer" containerID="9705eaa320085d8fe305beb20f9205e88fe8cd64ed0a9d9c13b667a162cf3af4"
Nov 02 18:48:56.745660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.745617    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"]
Nov 02 18:48:56.794468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.794437    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"]
Nov 02 18:48:56.850941 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.850913    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:56.907364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.907332    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"]
Nov 02 18:48:56.910390 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:56.910362    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-gbzt9"]
Nov 02 18:48:57.195507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:57.195472    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:57.383337 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:57.383305    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" path="/var/lib/kubelet/pods/af9e95d7-9f83-4388-b59b-7e14307e88ba/volumes"
Nov 02 18:48:57.538799 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:57.538714    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:58.158663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:58.158605    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:58.527143 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:48:58.527053    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:48:59.062054 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:48:59.062018    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:49:01.875011 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:01.874931    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:02.168338 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:02.168249    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:03.078431 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:03.078394    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:03.388520 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:03.388485    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:04.348419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:04.348386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:04.681860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:04.681811    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:07.371584 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:07.371552    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:08.321463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:08.321423    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"]
Nov 02 18:49:08.438684 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:08.438651    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"]
Nov 02 18:49:08.743061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:08.743029    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:09.040172 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:09.040090    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:09.404165 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:09.404134    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:09.922044 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:09.922012    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:11.768092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:11.768054    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" podUID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" containerName="main" containerID="cri-o://fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410" gracePeriod=45
Nov 02 18:49:11.814619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:11.814581    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" podUID="ef20ad96-c209-4313-8a19-02c3d9b5d999" containerName="main" containerID="cri-o://e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a" gracePeriod=45
Nov 02 18:49:11.979083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:11.979023    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" podUID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused"
Nov 02 18:49:12.090219 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.090188    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 18:49:12.111571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.111549    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 18:49:12.161857 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.161822    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm\") pod \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") "
Nov 02 18:49:12.162065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.161976    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir\") pod \"ef20ad96-c209-4313-8a19-02c3d9b5d999\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") "
Nov 02 18:49:12.162065 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162041    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home\") pod \"ef20ad96-c209-4313-8a19-02c3d9b5d999\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") "
Nov 02 18:49:12.162197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162106    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache\") pod \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") "
Nov 02 18:49:12.162197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162137    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir\") pod \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") "
Nov 02 18:49:12.162197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162179    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache\") pod \"ef20ad96-c209-4313-8a19-02c3d9b5d999\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") "
Nov 02 18:49:12.162350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162215    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm\") pod \"ef20ad96-c209-4313-8a19-02c3d9b5d999\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") "
Nov 02 18:49:12.162350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162243    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home\") pod \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") "
Nov 02 18:49:12.162350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162273    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr\" (UniqueName: \"kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr\") pod \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") "
Nov 02 18:49:12.162350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162333    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz\" (UniqueName: \"kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz\") pod \"ef20ad96-c209-4313-8a19-02c3d9b5d999\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") "
Nov 02 18:49:12.162548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162412    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs\") pod \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\" (UID: \"5f24edb2-2853-4d83-ad30-4d4d64d7823c\") "
Nov 02 18:49:12.162548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162452    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs\") pod \"ef20ad96-c209-4313-8a19-02c3d9b5d999\" (UID: \"ef20ad96-c209-4313-8a19-02c3d9b5d999\") "
Nov 02 18:49:12.162548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162492    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache" pod "5f24edb2-2853-4d83-ad30-4d4d64d7823c" (UID: "5f24edb2-2853-4d83-ad30-4d4d64d7823c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.162548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162511    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir" pod "5f24edb2-2853-4d83-ad30-4d4d64d7823c" (UID: "5f24edb2-2853-4d83-ad30-4d4d64d7823c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.162785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162646    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home" pod "5f24edb2-2853-4d83-ad30-4d4d64d7823c" (UID: "5f24edb2-2853-4d83-ad30-4d4d64d7823c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.162785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162622    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home" pod "ef20ad96-c209-4313-8a19-02c3d9b5d999" (UID: "ef20ad96-c209-4313-8a19-02c3d9b5d999"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.162785 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162720    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache" pod "ef20ad96-c209-4313-8a19-02c3d9b5d999" (UID: "ef20ad96-c209-4313-8a19-02c3d9b5d999"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.163095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.162976    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.163095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.163004    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.163095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.163045    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.163095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.163061    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.163095 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.163073    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.163675 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.163148    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir" pod "ef20ad96-c209-4313-8a19-02c3d9b5d999" (UID: "ef20ad96-c209-4313-8a19-02c3d9b5d999"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.165097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.165066    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs" pod "5f24edb2-2853-4d83-ad30-4d4d64d7823c" (UID: "5f24edb2-2853-4d83-ad30-4d4d64d7823c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:12.165097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.165079    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm" pod "5f24edb2-2853-4d83-ad30-4d4d64d7823c" (UID: "5f24edb2-2853-4d83-ad30-4d4d64d7823c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.165263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.165071    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm" pod "ef20ad96-c209-4313-8a19-02c3d9b5d999" (UID: "ef20ad96-c209-4313-8a19-02c3d9b5d999"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:12.165263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.165161    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz" pod "ef20ad96-c209-4313-8a19-02c3d9b5d999" (UID: "ef20ad96-c209-4313-8a19-02c3d9b5d999"). InnerVolumeSpecName "kube-api-access-v9sgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:12.165418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.165402    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr" pod "5f24edb2-2853-4d83-ad30-4d4d64d7823c" (UID: "5f24edb2-2853-4d83-ad30-4d4d64d7823c"). InnerVolumeSpecName "kube-api-access-gk5vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:12.165456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.165427    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs" pod "ef20ad96-c209-4313-8a19-02c3d9b5d999" (UID: "ef20ad96-c209-4313-8a19-02c3d9b5d999"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:12.264280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264245    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f24edb2-2853-4d83-ad30-4d4d64d7823c-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.264280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264274    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef20ad96-c209-4313-8a19-02c3d9b5d999-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.264280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264283    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f24edb2-2853-4d83-ad30-4d4d64d7823c-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.264280 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264292    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.264701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264300    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef20ad96-c209-4313-8a19-02c3d9b5d999-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.264701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264309    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5vr\" (UniqueName: \"kubernetes.io/projected/5f24edb2-2853-4d83-ad30-4d4d64d7823c-kube-api-access-gk5vr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.264701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.264319    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9sgz\" (UniqueName: \"kubernetes.io/projected/ef20ad96-c209-4313-8a19-02c3d9b5d999-kube-api-access-v9sgz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:12.850179 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.850141    4476 generic.go:356] "Generic (PLEG): container finished" podID="ef20ad96-c209-4313-8a19-02c3d9b5d999" containerID="e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a" exitCode=0
Nov 02 18:49:12.850691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.850192    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"
Nov 02 18:49:12.850691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.850224    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" event={"ID":"ef20ad96-c209-4313-8a19-02c3d9b5d999","Type":"ContainerDied","Data":"e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a"}
Nov 02 18:49:12.850691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.850263    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn" event={"ID":"ef20ad96-c209-4313-8a19-02c3d9b5d999","Type":"ContainerDied","Data":"29987ca1f6ddd223640647163a78ebbcf1342f361bdeed387cbf942134430e2c"}
Nov 02 18:49:12.850691 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.850280    4476 scope.go:122] "RemoveContainer" containerID="e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a"
Nov 02 18:49:12.851832 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.851808    4476 generic.go:356] "Generic (PLEG): container finished" podID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" containerID="fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410" exitCode=0
Nov 02 18:49:12.851938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.851844    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"
Nov 02 18:49:12.851938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.851876    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" event={"ID":"5f24edb2-2853-4d83-ad30-4d4d64d7823c","Type":"ContainerDied","Data":"fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410"}
Nov 02 18:49:12.851938 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.851904    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78" event={"ID":"5f24edb2-2853-4d83-ad30-4d4d64d7823c","Type":"ContainerDied","Data":"86b74ee5a14193788ebd3df397029065d4b3095c8f7077dccd1f5ba54ff8847d"}
Nov 02 18:49:12.862468 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.862351    4476 scope.go:122] "RemoveContainer" containerID="e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a"
Nov 02 18:49:12.862717 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:12.862693    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a\": container with ID starting with e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a not found: ID does not exist" containerID="e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a"
Nov 02 18:49:12.862794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.862731    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a"} err="failed to get container status \"e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a\": rpc error: code = NotFound desc = could not find container \"e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a\": container with ID starting with e6614c0d1c3d11a7a4cf423b5b9be00c694c7c784b7e18c19056172d8004d75a not found: ID does not exist"
Nov 02 18:49:12.862794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.862756    4476 scope.go:122] "RemoveContainer" containerID="fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410"
Nov 02 18:49:12.871873 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.871857    4476 scope.go:122] "RemoveContainer" containerID="fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410"
Nov 02 18:49:12.872202 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:12.872176    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410\": container with ID starting with fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410 not found: ID does not exist" containerID="fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410"
Nov 02 18:49:12.872279 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:12.872211    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410"} err="failed to get container status \"fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410\": rpc error: code = NotFound desc = could not find container \"fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410\": container with ID starting with fe264054b8e3792aa5cdd8eaf139fd613617b68b6c9fe41d64c4cc10b0fa0410 not found: ID does not exist"
Nov 02 18:49:13.012415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.012383    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"]
Nov 02 18:49:13.049639 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.049544    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:13.061660 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.061613    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/premium-w4-4ad655-kserve-769756558c-7wr78"]
Nov 02 18:49:13.124748 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.124714    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"]
Nov 02 18:49:13.127614 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.127591    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ad655/simulator-w4-4ad655-kserve-5f6c9ffc94-pmwgn"]
Nov 02 18:49:13.322331 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.322252    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz" podUID="f54391c3-dd1f-4e0f-b407-e91611f79e93" containerName="payload-processing" containerID="cri-o://41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d" gracePeriod=25
Nov 02 18:49:13.358257 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.358221    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:13.385582 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.385546    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" path="/var/lib/kubelet/pods/5f24edb2-2853-4d83-ad30-4d4d64d7823c/volumes"
Nov 02 18:49:13.385990 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.385976    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef20ad96-c209-4313-8a19-02c3d9b5d999" path="/var/lib/kubelet/pods/ef20ad96-c209-4313-8a19-02c3d9b5d999/volumes"
Nov 02 18:49:13.439813 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.439699    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm" podUID="3fcae5a9-0fdc-4c39-8025-e5f714fbf883" containerName="payload-pre-processing" containerID="cri-o://572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8" gracePeriod=25
Nov 02 18:49:13.610150 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.610123    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 18:49:13.680378 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.680343    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6\" (UniqueName: \"kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6\") pod \"f54391c3-dd1f-4e0f-b407-e91611f79e93\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") "
Nov 02 18:49:13.680591 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.680399    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume\") pod \"f54391c3-dd1f-4e0f-b407-e91611f79e93\" (UID: \"f54391c3-dd1f-4e0f-b407-e91611f79e93\") "
Nov 02 18:49:13.680991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.680963    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume" pod "f54391c3-dd1f-4e0f-b407-e91611f79e93" (UID: "f54391c3-dd1f-4e0f-b407-e91611f79e93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:13.683853 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.683823    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6" pod "f54391c3-dd1f-4e0f-b407-e91611f79e93" (UID: "f54391c3-dd1f-4e0f-b407-e91611f79e93"). InnerVolumeSpecName "kube-api-access-fncv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:13.687042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.687013    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fncv6\" (UniqueName: \"kubernetes.io/projected/f54391c3-dd1f-4e0f-b407-e91611f79e93-kube-api-access-fncv6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:13.687042 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.687041    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54391c3-dd1f-4e0f-b407-e91611f79e93-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:13.711548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.711520    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 18:49:13.787531 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.787497    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp\" (UniqueName: \"kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp\") pod \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") "
Nov 02 18:49:13.787704 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.787556    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume\") pod \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\" (UID: \"3fcae5a9-0fdc-4c39-8025-e5f714fbf883\") "
Nov 02 18:49:13.788059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.788030    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume" pod "3fcae5a9-0fdc-4c39-8025-e5f714fbf883" (UID: "3fcae5a9-0fdc-4c39-8025-e5f714fbf883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:13.790008 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.789975    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp" pod "3fcae5a9-0fdc-4c39-8025-e5f714fbf883" (UID: "3fcae5a9-0fdc-4c39-8025-e5f714fbf883"). InnerVolumeSpecName "kube-api-access-lgsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:13.863747 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.863709    4476 generic.go:356] "Generic (PLEG): container finished" podID="f54391c3-dd1f-4e0f-b407-e91611f79e93" containerID="41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d" exitCode=0
Nov 02 18:49:13.864273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.863753    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"
Nov 02 18:49:13.864273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.863792    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz" event={"ID":"f54391c3-dd1f-4e0f-b407-e91611f79e93","Type":"ContainerDied","Data":"41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d"}
Nov 02 18:49:13.864273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.863842    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz" event={"ID":"f54391c3-dd1f-4e0f-b407-e91611f79e93","Type":"ContainerDied","Data":"2e921309ff853685c921d4538f641f89b131cb348ee320ca14aee4141ad892a7"}
Nov 02 18:49:13.864273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.863874    4476 scope.go:122] "RemoveContainer" containerID="41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d"
Nov 02 18:49:13.866517 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.866489    4476 generic.go:356] "Generic (PLEG): container finished" podID="3fcae5a9-0fdc-4c39-8025-e5f714fbf883" containerID="572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8" exitCode=0
Nov 02 18:49:13.866681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.866545    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm" event={"ID":"3fcae5a9-0fdc-4c39-8025-e5f714fbf883","Type":"ContainerDied","Data":"572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8"}
Nov 02 18:49:13.866681 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.866587    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm" event={"ID":"3fcae5a9-0fdc-4c39-8025-e5f714fbf883","Type":"ContainerDied","Data":"55ad3543276464afa9205547ed916c220f62eae1eadbb2d2571f078c0f55ae02"}
Nov 02 18:49:13.866860 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.866790    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"
Nov 02 18:49:13.878533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.878510    4476 scope.go:122] "RemoveContainer" containerID="41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d"
Nov 02 18:49:13.879013 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:13.878896    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d\": container with ID starting with 41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d not found: ID does not exist" containerID="41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d"
Nov 02 18:49:13.879013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.878934    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d"} err="failed to get container status \"41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d\": rpc error: code = NotFound desc = could not find container \"41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d\": container with ID starting with 41a8c3090d06af1d05ebf31a7d2f5ddc412003bc588c5d7b39bc2cb52c9e4b5d not found: ID does not exist"
Nov 02 18:49:13.879013 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.878957    4476 scope.go:122] "RemoveContainer" containerID="572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8"
Nov 02 18:49:13.888817 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.888792    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgsjp\" (UniqueName: \"kubernetes.io/projected/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-kube-api-access-lgsjp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:13.888940 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.888820    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fcae5a9-0fdc-4c39-8025-e5f714fbf883-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:13.890908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.890887    4476 scope.go:122] "RemoveContainer" containerID="572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8"
Nov 02 18:49:13.891236 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:13.891204    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8\": container with ID starting with 572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8 not found: ID does not exist" containerID="572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8"
Nov 02 18:49:13.891297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:13.891250    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8"} err="failed to get container status \"572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8\": rpc error: code = NotFound desc = could not find container \"572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8\": container with ID starting with 572b44de2d5cd77f67eee5a3fa29c9a3343056e1877b0a2b6b07813b150392f8 not found: ID does not exist"
Nov 02 18:49:14.026795 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.026762    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"]
Nov 02 18:49:14.029887 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.029860    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ad655-5b4569c8b7-wx6mm"]
Nov 02 18:49:14.138736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.138641    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"]
Nov 02 18:49:14.141582 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.141555    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ad655-75bfbfd57-c8sjz"]
Nov 02 18:49:14.370723 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.370689    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:14.671023 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.670989    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:14.990083 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:14.989992    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:15.385134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:15.385097    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fcae5a9-0fdc-4c39-8025-e5f714fbf883" path="/var/lib/kubelet/pods/3fcae5a9-0fdc-4c39-8025-e5f714fbf883/volumes"
Nov 02 18:49:15.385672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:15.385620    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54391c3-dd1f-4e0f-b407-e91611f79e93" path="/var/lib/kubelet/pods/f54391c3-dd1f-4e0f-b407-e91611f79e93/volumes"
Nov 02 18:49:15.670736 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:15.670655    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:16.004469 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:16.004386    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:16.355912 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:16.355833    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:16.739086 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:16.739050    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:19.550889 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:19.550852    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:21.921196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:21.921166    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:22.260438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:22.260353    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:22.652679 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:22.652607    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:23.639955 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:23.639738    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:24.968099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:24.968066    4476 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log" limit=1024
Nov 02 18:49:28.905078 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:28.905045    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"]
Nov 02 18:49:28.905450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:28.905247    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" podUID="c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" containerName="maas-api" containerID="cri-o://2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e" gracePeriod=30
Nov 02 18:49:29.067217 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:29.067168    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:49:29.242206 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.242161    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:49:29.262408 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.262371    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp\") pod \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") "
Nov 02 18:49:29.262593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.262514    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca\") pod \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") "
Nov 02 18:49:29.262593 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.262563    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\") pod \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") "
Nov 02 18:49:29.262756 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.262647    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs\") pod \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\" (UID: \"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a\") "
Nov 02 18:49:29.263235 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.263096    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca" pod "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" (UID: "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:29.265421 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.265381    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs" pod "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" (UID: "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:29.265670 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.265606    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls" pod "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" (UID: "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:29.265843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.265823    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp" pod "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" (UID: "c6ad59c1-566a-4f0f-99d4-5e3df926ca5a"). InnerVolumeSpecName "kube-api-access-cffpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:29.365129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.365101    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.365129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.365130    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.365375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.365144    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.365375 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.365156    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a-kube-api-access-cffpp\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:29.997565 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.997528    4476 generic.go:356] "Generic (PLEG): container finished" podID="c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" containerID="2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e" exitCode=0
Nov 02 18:49:29.998010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.997590    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"
Nov 02 18:49:29.998010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.997616    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" event={"ID":"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a","Type":"ContainerDied","Data":"2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e"}
Nov 02 18:49:29.998010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.997689    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g" event={"ID":"c6ad59c1-566a-4f0f-99d4-5e3df926ca5a","Type":"ContainerDied","Data":"e814a48b55268ffddca1aa55ae4cb0f8c6265843880958c956cdd315f43b580a"}
Nov 02 18:49:29.998010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:29.997719    4476 scope.go:122] "RemoveContainer" containerID="2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e"
Nov 02 18:49:30.007668 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:30.007650    4476 scope.go:122] "RemoveContainer" containerID="2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e"
Nov 02 18:49:30.007977 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:30.007950    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e\": container with ID starting with 2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e not found: ID does not exist" containerID="2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e"
Nov 02 18:49:30.008091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:30.007984    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e"} err="failed to get container status \"2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e\": rpc error: code = NotFound desc = could not find container \"2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e\": container with ID starting with 2517b6f3830a83c2e92897db37dfc0db72ab1da3e96c09da99a35626efee382e not found: ID does not exist"
Nov 02 18:49:30.162455 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:30.162424    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"]
Nov 02 18:49:30.167214 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:30.167189    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4ecad8-59d99cc654-8hv4g"]
Nov 02 18:49:31.384433 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:31.384398    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" path="/var/lib/kubelet/pods/c6ad59c1-566a-4f0f-99d4-5e3df926ca5a/volumes"
Nov 02 18:49:33.914369 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:33.914335    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"]
Nov 02 18:49:33.914868 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:33.914540    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" containerID="cri-o://009ac9804e0a52d5f3d3679190158abb25478a7716507e8ed170f1b6aeff4fa4" gracePeriod=30
Nov 02 18:49:37.226412 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:37.226372    4476 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:49:37.226828 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:37.226439    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:49:39.067742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.067699    4476 generic.go:356] "Generic (PLEG): container finished" podID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerID="009ac9804e0a52d5f3d3679190158abb25478a7716507e8ed170f1b6aeff4fa4" exitCode=0
Nov 02 18:49:39.068372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.067772    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" event={"ID":"03a85290-1c16-4c6b-8cfe-89922ebac737","Type":"ContainerDied","Data":"009ac9804e0a52d5f3d3679190158abb25478a7716507e8ed170f1b6aeff4fa4"}
Nov 02 18:49:39.068372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.067851    4476 scope.go:122] "RemoveContainer" containerID="b763fac65251504cace4a9113ca2c45787cf534071dac5e029e6e0ea37cf3151"
Nov 02 18:49:39.186620 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.186554    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:49:39.221609 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.221580    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"]
Nov 02 18:49:39.272019 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.271987    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.272222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272044    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.272222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272066    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.272222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272136    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.272222 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272193    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.272457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272231    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.272457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272260    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.273106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272866    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.273106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.272940    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo\") pod \"03a85290-1c16-4c6b-8cfe-89922ebac737\" (UID: \"03a85290-1c16-4c6b-8cfe-89922ebac737\") "
Nov 02 18:49:39.273106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273024    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:39.273106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273062    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:39.273106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273086    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:39.273492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273170    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:39.273492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273353    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:49:39.273492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273463    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.273492 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273485    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a85290-1c16-4c6b-8cfe-89922ebac737-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.273710 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273502    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.273710 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273516    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.273710 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.273530    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.274881 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.274851    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:39.275121 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.275096    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:39.275418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.275400    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "kube-api-access-x4vz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:39.275521 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.275507    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo" pod "03a85290-1c16-4c6b-8cfe-89922ebac737" (UID: "03a85290-1c16-4c6b-8cfe-89922ebac737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:49:39.374842 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.374809    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vz8\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-kube-api-access-x4vz8\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.374842 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.374840    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.374842 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.374850    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:39.375197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:39.374860    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a85290-1c16-4c6b-8cfe-89922ebac737-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:40.075739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:40.075696    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"
Nov 02 18:49:40.075739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:40.075731    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8" event={"ID":"03a85290-1c16-4c6b-8cfe-89922ebac737","Type":"ContainerDied","Data":"40e593c9d4e4b828cfc717a588740fbea9399d8402aff6ecc151a3ace503e7ac"}
Nov 02 18:49:40.076300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:40.075790    4476 scope.go:122] "RemoveContainer" containerID="009ac9804e0a52d5f3d3679190158abb25478a7716507e8ed170f1b6aeff4fa4"
Nov 02 18:49:40.238883 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:40.238834    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"]
Nov 02 18:49:40.288140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:40.288102    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4ecad8-openshift-default-84b9dd7bb9-mwll8"]
Nov 02 18:49:41.384466 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:41.384428    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" path="/var/lib/kubelet/pods/03a85290-1c16-4c6b-8cfe-89922ebac737/volumes"
Nov 02 18:49:49.907269 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:49.907233    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"]
Nov 02 18:49:50.014423 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:50.014383    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"]
Nov 02 18:49:54.242841 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.242779    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" podUID="8d97ceb9-8398-4bd7-aea8-174135bcd543" containerName="main" containerID="cri-o://a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65" gracePeriod=45
Nov 02 18:49:54.502263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.502183    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 18:49:54.637476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637373    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache\") pod \"8d97ceb9-8398-4bd7-aea8-174135bcd543\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") "
Nov 02 18:49:54.637476 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637434    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665\" (UniqueName: \"kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665\") pod \"8d97ceb9-8398-4bd7-aea8-174135bcd543\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") "
Nov 02 18:49:54.637801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs\") pod \"8d97ceb9-8398-4bd7-aea8-174135bcd543\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") "
Nov 02 18:49:54.637801 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637537    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir\") pod \"8d97ceb9-8398-4bd7-aea8-174135bcd543\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") "
Nov 02 18:49:54.637934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637815    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache" pod "8d97ceb9-8398-4bd7-aea8-174135bcd543" (UID: "8d97ceb9-8398-4bd7-aea8-174135bcd543"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.637934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637821    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir" pod "8d97ceb9-8398-4bd7-aea8-174135bcd543" (UID: "8d97ceb9-8398-4bd7-aea8-174135bcd543"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.637934 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.637840    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home\") pod \"8d97ceb9-8398-4bd7-aea8-174135bcd543\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") "
Nov 02 18:49:54.638112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.638019    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm\") pod \"8d97ceb9-8398-4bd7-aea8-174135bcd543\" (UID: \"8d97ceb9-8398-4bd7-aea8-174135bcd543\") "
Nov 02 18:49:54.638112 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.638051    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home" pod "8d97ceb9-8398-4bd7-aea8-174135bcd543" (UID: "8d97ceb9-8398-4bd7-aea8-174135bcd543"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.638571 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.638479    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.638701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.638621    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.638701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.638653    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.640049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.640027    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665" pod "8d97ceb9-8398-4bd7-aea8-174135bcd543" (UID: "8d97ceb9-8398-4bd7-aea8-174135bcd543"). InnerVolumeSpecName "kube-api-access-xv665". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:49:54.640181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.640161    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs" pod "8d97ceb9-8398-4bd7-aea8-174135bcd543" (UID: "8d97ceb9-8398-4bd7-aea8-174135bcd543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:49:54.640181 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.640159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm" pod "8d97ceb9-8398-4bd7-aea8-174135bcd543" (UID: "8d97ceb9-8398-4bd7-aea8-174135bcd543"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:49:54.739425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.739388    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv665\" (UniqueName: \"kubernetes.io/projected/8d97ceb9-8398-4bd7-aea8-174135bcd543-kube-api-access-xv665\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.739425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.739422    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d97ceb9-8398-4bd7-aea8-174135bcd543-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.739425 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.739432    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d97ceb9-8398-4bd7-aea8-174135bcd543-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:49:54.908576 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:54.908522    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerName="payload-processing" containerID="cri-o://3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff" gracePeriod=25
Nov 02 18:49:55.015243 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.015201    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerName="payload-pre-processing" containerID="cri-o://6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347" gracePeriod=25
Nov 02 18:49:55.193240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.193143    4476 generic.go:356] "Generic (PLEG): container finished" podID="8d97ceb9-8398-4bd7-aea8-174135bcd543" containerID="a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65" exitCode=0
Nov 02 18:49:55.193240 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.193195    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"
Nov 02 18:49:55.193446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.193233    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" event={"ID":"8d97ceb9-8398-4bd7-aea8-174135bcd543","Type":"ContainerDied","Data":"a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65"}
Nov 02 18:49:55.193446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.193290    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n" event={"ID":"8d97ceb9-8398-4bd7-aea8-174135bcd543","Type":"ContainerDied","Data":"a420d5268a3720749d5ea3d5ef5040fa1ae9076c1814e76e51d3cde3f1a02332"}
Nov 02 18:49:55.193446 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.193313    4476 scope.go:122] "RemoveContainer" containerID="a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65"
Nov 02 18:49:55.205496 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.205474    4476 scope.go:122] "RemoveContainer" containerID="a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65"
Nov 02 18:49:55.205773 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:55.205755    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65\": container with ID starting with a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65 not found: ID does not exist" containerID="a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65"
Nov 02 18:49:55.205831 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.205781    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65"} err="failed to get container status \"a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65\": rpc error: code = NotFound desc = could not find container \"a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65\": container with ID starting with a104bbeeee4629286d56485aff6bfab5cf40eadc961718dd40d1f319d1ddbe65 not found: ID does not exist"
Nov 02 18:49:55.356213 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.356170    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"]
Nov 02 18:49:55.405661 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:55.405616    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-w1-dc5689-kserve-66fdc755cc-wf92n"]
Nov 02 18:49:57.382321 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:49:57.382286    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d97ceb9-8398-4bd7-aea8-174135bcd543" path="/var/lib/kubelet/pods/8d97ceb9-8398-4bd7-aea8-174135bcd543/volumes"
Nov 02 18:49:59.062033 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:49:59.061989    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:50:00.948092 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:00.948054    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"]
Nov 02 18:50:00.995322 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:00.995290    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"]
Nov 02 18:50:01.111751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:01.111710    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:01.111751 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:01.111743    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:02.017196 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.017165    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"]
Nov 02 18:50:02.017737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.017377    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" podUID="60d64f4b-7a32-4013-96dd-415fcd0d5e2f" containerName="maas-api" containerID="cri-o://a0275eb17f4072d1db5dc60be5af1c33f9dbfd44d12d03e00e65295421a298cb" gracePeriod=30
Nov 02 18:50:02.270300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.269972    4476 generic.go:356] "Generic (PLEG): container finished" podID="60d64f4b-7a32-4013-96dd-415fcd0d5e2f" containerID="a0275eb17f4072d1db5dc60be5af1c33f9dbfd44d12d03e00e65295421a298cb" exitCode=0
Nov 02 18:50:02.270300 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.270126    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" event={"ID":"60d64f4b-7a32-4013-96dd-415fcd0d5e2f","Type":"ContainerDied","Data":"a0275eb17f4072d1db5dc60be5af1c33f9dbfd44d12d03e00e65295421a298cb"}
Nov 02 18:50:02.319985 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.319954    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:50:02.419498 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.419460    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf\" (UniqueName: \"kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf\") pod \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") "
Nov 02 18:50:02.419729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.419555    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs\") pod \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") "
Nov 02 18:50:02.419729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.419584    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca\") pod \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") "
Nov 02 18:50:02.419729 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.419656    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\") pod \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\" (UID: \"60d64f4b-7a32-4013-96dd-415fcd0d5e2f\") "
Nov 02 18:50:02.422233 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.421155    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca" pod "60d64f4b-7a32-4013-96dd-415fcd0d5e2f" (UID: "60d64f4b-7a32-4013-96dd-415fcd0d5e2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:02.427650 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.424157    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf" pod "60d64f4b-7a32-4013-96dd-415fcd0d5e2f" (UID: "60d64f4b-7a32-4013-96dd-415fcd0d5e2f"). InnerVolumeSpecName "kube-api-access-7pvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:02.441652 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.438745    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls" pod "60d64f4b-7a32-4013-96dd-415fcd0d5e2f" (UID: "60d64f4b-7a32-4013-96dd-415fcd0d5e2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:02.464241 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.464201    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs" pod "60d64f4b-7a32-4013-96dd-415fcd0d5e2f" (UID: "60d64f4b-7a32-4013-96dd-415fcd0d5e2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:02.521775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.521621    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvlf\" (UniqueName: \"kubernetes.io/projected/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-kube-api-access-7pvlf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:02.521775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.521674    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:02.521775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.521689    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:02.521775 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:02.521702    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d64f4b-7a32-4013-96dd-415fcd0d5e2f-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:03.279008 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:03.278975    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"
Nov 02 18:50:03.279561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:03.279007    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c" event={"ID":"60d64f4b-7a32-4013-96dd-415fcd0d5e2f","Type":"ContainerDied","Data":"1bdd5ec9dad30b59916f54ad3dfb73e3d8a6aeb7584381751bad0b8c22a5a590"}
Nov 02 18:50:03.279561 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:03.279077    4476 scope.go:122] "RemoveContainer" containerID="a0275eb17f4072d1db5dc60be5af1c33f9dbfd44d12d03e00e65295421a298cb"
Nov 02 18:50:03.443185 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:03.443152    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"]
Nov 02 18:50:03.491344 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:03.491303    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c82472-588f797dc4-ml72c"]
Nov 02 18:50:05.384107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:05.384076    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d64f4b-7a32-4013-96dd-415fcd0d5e2f" path="/var/lib/kubelet/pods/60d64f4b-7a32-4013-96dd-415fcd0d5e2f/volumes"
Nov 02 18:50:11.112364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:11.112323    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:11.112904 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:11.112332    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:15.175357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.175299    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 18:50:15.265067 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.265038    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg\" (UniqueName: \"kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg\") pod \"9c22053b-aceb-4f84-a364-78c65b651ac8\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") "
Nov 02 18:50:15.265237 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.265121    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume\") pod \"9c22053b-aceb-4f84-a364-78c65b651ac8\" (UID: \"9c22053b-aceb-4f84-a364-78c65b651ac8\") "
Nov 02 18:50:15.265535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.265514    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume" pod "9c22053b-aceb-4f84-a364-78c65b651ac8" (UID: "9c22053b-aceb-4f84-a364-78c65b651ac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:15.267251 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.267224    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg" pod "9c22053b-aceb-4f84-a364-78c65b651ac8" (UID: "9c22053b-aceb-4f84-a364-78c65b651ac8"). InnerVolumeSpecName "kube-api-access-lf2pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:15.280282 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.280250    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 18:50:15.365532 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365496    4476 generic.go:356] "Generic (PLEG): container finished" podID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerID="6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347" exitCode=0
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365579    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365589    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" event={"ID":"6fc33efd-3688-4ef5-8298-00f8d0e7554b","Type":"ContainerDied","Data":"6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347"}
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365714    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k" event={"ID":"6fc33efd-3688-4ef5-8298-00f8d0e7554b","Type":"ContainerDied","Data":"94056df4ce88c38e6c113f92a1f79a2879ec0ca835492e5d2bb611151b5e0194"}
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365729    4476 scope.go:122] "RemoveContainer" containerID="6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347"
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365809    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume\") pod \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") "
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.365899    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc\" (UniqueName: \"kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc\") pod \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\" (UID: \"6fc33efd-3688-4ef5-8298-00f8d0e7554b\") "
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.366217    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume" pod "6fc33efd-3688-4ef5-8298-00f8d0e7554b" (UID: "6fc33efd-3688-4ef5-8298-00f8d0e7554b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.366294    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2pg\" (UniqueName: \"kubernetes.io/projected/9c22053b-aceb-4f84-a364-78c65b651ac8-kube-api-access-lf2pg\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.366310    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c22053b-aceb-4f84-a364-78c65b651ac8-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.367489    4476 generic.go:356] "Generic (PLEG): container finished" podID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerID="3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff" exitCode=0
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.367533    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.367562    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" event={"ID":"9c22053b-aceb-4f84-a364-78c65b651ac8","Type":"ContainerDied","Data":"3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff"}
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.367601    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp" event={"ID":"9c22053b-aceb-4f84-a364-78c65b651ac8","Type":"ContainerDied","Data":"ea3357d63bfd359b845ce7d8534e00e842ee07dd203edf721b90b848fa591630"}
Nov 02 18:50:15.369610 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.368365    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc" pod "6fc33efd-3688-4ef5-8298-00f8d0e7554b" (UID: "6fc33efd-3688-4ef5-8298-00f8d0e7554b"). InnerVolumeSpecName "kube-api-access-zfqxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:15.375865 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.375847    4476 scope.go:122] "RemoveContainer" containerID="6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347"
Nov 02 18:50:15.376269 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:15.376244    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347\": container with ID starting with 6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347 not found: ID does not exist" containerID="6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347"
Nov 02 18:50:15.376366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.376277    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347"} err="failed to get container status \"6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347\": rpc error: code = NotFound desc = could not find container \"6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347\": container with ID starting with 6bcde2e1b31c93fe338929f0875936b0c209cc897f6b0fa30994149d02a40347 not found: ID does not exist"
Nov 02 18:50:15.376366 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.376295    4476 scope.go:122] "RemoveContainer" containerID="3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff"
Nov 02 18:50:15.385360 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.385335    4476 scope.go:122] "RemoveContainer" containerID="3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff"
Nov 02 18:50:15.385758 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:15.385732    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff\": container with ID starting with 3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff not found: ID does not exist" containerID="3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff"
Nov 02 18:50:15.385848 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.385765    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff"} err="failed to get container status \"3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff\": rpc error: code = NotFound desc = could not find container \"3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff\": container with ID starting with 3cb0738aeca97c889b09b5b0c540a16e412827460186f30ddc74d89988da40ff not found: ID does not exist"
Nov 02 18:50:15.468134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.467951    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc33efd-3688-4ef5-8298-00f8d0e7554b-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:15.468134 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.467988    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqxc\" (UniqueName: \"kubernetes.io/projected/6fc33efd-3688-4ef5-8298-00f8d0e7554b-kube-api-access-zfqxc\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:15.594507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.594470    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"]
Nov 02 18:50:15.605569 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.605533    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-dc5689-86b8d87699-2h4qp"]
Nov 02 18:50:15.783132 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.783056    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"]
Nov 02 18:50:15.791135 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.791106    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dc5689-6c8445955f-lhq4k"]
Nov 02 18:50:15.971976 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:15.971925    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5" podUID="f0018d34-408d-4cf5-8853-6b08d369fbbb" containerName="main" containerID="cri-o://46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742" gracePeriod=45
Nov 02 18:50:16.016049 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.016006    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8" podUID="44f022ae-9bf4-4ecd-a105-08b66edf63ee" containerName="main" containerID="cri-o://676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56" gracePeriod=45
Nov 02 18:50:16.288315 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.288294    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 18:50:16.298417 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.298393    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 18:50:16.375111 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.375079    4476 generic.go:356] "Generic (PLEG): container finished" podID="f0018d34-408d-4cf5-8853-6b08d369fbbb" containerID="46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742" exitCode=0
Nov 02 18:50:16.375284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.375122    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"
Nov 02 18:50:16.375284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.375159    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5" event={"ID":"f0018d34-408d-4cf5-8853-6b08d369fbbb","Type":"ContainerDied","Data":"46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742"}
Nov 02 18:50:16.375284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.375245    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5" event={"ID":"f0018d34-408d-4cf5-8853-6b08d369fbbb","Type":"ContainerDied","Data":"cfd3ba100b4767bf70a70f2365bd2c3d7879d4cd2ef420291c4f39fdad15e5c9"}
Nov 02 18:50:16.375284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.375267    4476 scope.go:122] "RemoveContainer" containerID="46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742"
Nov 02 18:50:16.377263 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377238    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs\") pod \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") "
Nov 02 18:50:16.377396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377292    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home\") pod \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") "
Nov 02 18:50:16.377396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377339    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache\") pod \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") "
Nov 02 18:50:16.377396 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377387    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir\") pod \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") "
Nov 02 18:50:16.377579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377417    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm\") pod \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") "
Nov 02 18:50:16.377579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377449    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng\" (UniqueName: \"kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng\") pod \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\" (UID: \"44f022ae-9bf4-4ecd-a105-08b66edf63ee\") "
Nov 02 18:50:16.377579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377484    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz\" (UniqueName: \"kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz\") pod \"f0018d34-408d-4cf5-8853-6b08d369fbbb\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") "
Nov 02 18:50:16.377579 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377510    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm\") pod \"f0018d34-408d-4cf5-8853-6b08d369fbbb\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") "
Nov 02 18:50:16.377834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377600    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs\") pod \"f0018d34-408d-4cf5-8853-6b08d369fbbb\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") "
Nov 02 18:50:16.377834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377666    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home" pod "44f022ae-9bf4-4ecd-a105-08b66edf63ee" (UID: "44f022ae-9bf4-4ecd-a105-08b66edf63ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.377834 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377725    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir\") pod \"f0018d34-408d-4cf5-8853-6b08d369fbbb\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") "
Nov 02 18:50:16.378430 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.378333    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir" pod "44f022ae-9bf4-4ecd-a105-08b66edf63ee" (UID: "44f022ae-9bf4-4ecd-a105-08b66edf63ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.379283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.379259    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache" pod "44f022ae-9bf4-4ecd-a105-08b66edf63ee" (UID: "44f022ae-9bf4-4ecd-a105-08b66edf63ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.379460 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.377809    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache\") pod \"f0018d34-408d-4cf5-8853-6b08d369fbbb\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") "
Nov 02 18:50:16.379552 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.379493    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home\") pod \"f0018d34-408d-4cf5-8853-6b08d369fbbb\" (UID: \"f0018d34-408d-4cf5-8853-6b08d369fbbb\") "
Nov 02 18:50:16.379656 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.379611    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir" pod "f0018d34-408d-4cf5-8853-6b08d369fbbb" (UID: "f0018d34-408d-4cf5-8853-6b08d369fbbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.379989 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.379963    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.380074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.379994    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.380074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.380008    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.380074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.380013    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache" pod "f0018d34-408d-4cf5-8853-6b08d369fbbb" (UID: "f0018d34-408d-4cf5-8853-6b08d369fbbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.380074 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.380025    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.380283 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.380229    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home" pod "f0018d34-408d-4cf5-8853-6b08d369fbbb" (UID: "f0018d34-408d-4cf5-8853-6b08d369fbbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.380984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.380952    4476 generic.go:356] "Generic (PLEG): container finished" podID="44f022ae-9bf4-4ecd-a105-08b66edf63ee" containerID="676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56" exitCode=0
Nov 02 18:50:16.381091 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.381072    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8" event={"ID":"44f022ae-9bf4-4ecd-a105-08b66edf63ee","Type":"ContainerDied","Data":"676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56"}
Nov 02 18:50:16.381153 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.381114    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8" event={"ID":"44f022ae-9bf4-4ecd-a105-08b66edf63ee","Type":"ContainerDied","Data":"518044bec8e90450aea4c96f3e21891c6d63dcd209b3e1a09b409efee1c522f7"}
Nov 02 18:50:16.381232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.381175    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"
Nov 02 18:50:16.382796 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.382776    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm" pod "f0018d34-408d-4cf5-8853-6b08d369fbbb" (UID: "f0018d34-408d-4cf5-8853-6b08d369fbbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.382956 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.382932    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng" pod "44f022ae-9bf4-4ecd-a105-08b66edf63ee" (UID: "44f022ae-9bf4-4ecd-a105-08b66edf63ee"). InnerVolumeSpecName "kube-api-access-5gxng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:16.389555 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.389161    4476 scope.go:122] "RemoveContainer" containerID="46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742"
Nov 02 18:50:16.390049 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:16.390019    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742\": container with ID starting with 46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742 not found: ID does not exist" containerID="46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742"
Nov 02 18:50:16.390139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.390077    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742"} err="failed to get container status \"46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742\": rpc error: code = NotFound desc = could not find container \"46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742\": container with ID starting with 46ce40a7038988ab037426a7e24f7cf99ce0e81cc11b24ae0985dfc91d5d6742 not found: ID does not exist"
Nov 02 18:50:16.390139 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.390103    4476 scope.go:122] "RemoveContainer" containerID="676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56"
Nov 02 18:50:16.391281 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.391235    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs" pod "44f022ae-9bf4-4ecd-a105-08b66edf63ee" (UID: "44f022ae-9bf4-4ecd-a105-08b66edf63ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:16.391524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.391493    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs" pod "f0018d34-408d-4cf5-8853-6b08d369fbbb" (UID: "f0018d34-408d-4cf5-8853-6b08d369fbbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:16.391984 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.391963    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm" pod "44f022ae-9bf4-4ecd-a105-08b66edf63ee" (UID: "44f022ae-9bf4-4ecd-a105-08b66edf63ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:16.392317 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.392294    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz" pod "f0018d34-408d-4cf5-8853-6b08d369fbbb" (UID: "f0018d34-408d-4cf5-8853-6b08d369fbbb"). InnerVolumeSpecName "kube-api-access-hq8wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:16.399917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.399896    4476 scope.go:122] "RemoveContainer" containerID="676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56"
Nov 02 18:50:16.400225 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:16.400204    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56\": container with ID starting with 676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56 not found: ID does not exist" containerID="676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56"
Nov 02 18:50:16.400277 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.400235    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56"} err="failed to get container status \"676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56\": rpc error: code = NotFound desc = could not find container \"676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56\": container with ID starting with 676963f9e9fcffd28de9f65edd1cc7bd22f93a655d9f03d8fcf0266a9a19be56 not found: ID does not exist"
Nov 02 18:50:16.481061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481015    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481047    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481061 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481062    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f022ae-9bf4-4ecd-a105-08b66edf63ee-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481073    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f022ae-9bf4-4ecd-a105-08b66edf63ee-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481087    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxng\" (UniqueName: \"kubernetes.io/projected/44f022ae-9bf4-4ecd-a105-08b66edf63ee-kube-api-access-5gxng\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481101    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8wz\" (UniqueName: \"kubernetes.io/projected/f0018d34-408d-4cf5-8853-6b08d369fbbb-kube-api-access-hq8wz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481112    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0018d34-408d-4cf5-8853-6b08d369fbbb-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.481312 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.481122    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0018d34-408d-4cf5-8853-6b08d369fbbb-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:16.800699 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.800662    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"]
Nov 02 18:50:16.803464 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.803440    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-2-w0-ceb1ae-kserve-74765c6785-qxqk5"]
Nov 02 18:50:16.915820 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.915757    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"]
Nov 02 18:50:16.921176 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:16.921141    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/distinct-w0-ceb1ae-kserve-7955f8559d-bk9l8"]
Nov 02 18:50:17.387170 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:17.385996    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f022ae-9bf4-4ecd-a105-08b66edf63ee" path="/var/lib/kubelet/pods/44f022ae-9bf4-4ecd-a105-08b66edf63ee/volumes"
Nov 02 18:50:17.394268 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:17.394215    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" path="/var/lib/kubelet/pods/6fc33efd-3688-4ef5-8298-00f8d0e7554b/volumes"
Nov 02 18:50:17.395027 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:17.395006    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" path="/var/lib/kubelet/pods/9c22053b-aceb-4f84-a364-78c65b651ac8/volumes"
Nov 02 18:50:17.397685 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:17.395690    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0018d34-408d-4cf5-8853-6b08d369fbbb" path="/var/lib/kubelet/pods/f0018d34-408d-4cf5-8853-6b08d369fbbb/volumes"
Nov 02 18:50:23.646273 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:23.646234    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"]
Nov 02 18:50:23.646896 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:23.646513    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" podUID="7016d668-cd4b-4860-9908-2a97e39167a1" containerName="maas-api" containerID="cri-o://2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02" gracePeriod=30
Nov 02 18:50:23.971480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:23.971449    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 18:50:24.062400 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.062363    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca\") pod \"7016d668-cd4b-4860-9908-2a97e39167a1\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") "
Nov 02 18:50:24.062643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.062431    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs\") pod \"7016d668-cd4b-4860-9908-2a97e39167a1\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") "
Nov 02 18:50:24.062643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.062523    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\") pod \"7016d668-cd4b-4860-9908-2a97e39167a1\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") "
Nov 02 18:50:24.062643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.062603    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr\" (UniqueName: \"kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr\") pod \"7016d668-cd4b-4860-9908-2a97e39167a1\" (UID: \"7016d668-cd4b-4860-9908-2a97e39167a1\") "
Nov 02 18:50:24.062843 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.062803    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca" pod "7016d668-cd4b-4860-9908-2a97e39167a1" (UID: "7016d668-cd4b-4860-9908-2a97e39167a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:24.063017 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.062992    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7016d668-cd4b-4860-9908-2a97e39167a1-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:24.065913 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.065891    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr" pod "7016d668-cd4b-4860-9908-2a97e39167a1" (UID: "7016d668-cd4b-4860-9908-2a97e39167a1"). InnerVolumeSpecName "kube-api-access-x9jcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:24.066518 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.066491    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls" pod "7016d668-cd4b-4860-9908-2a97e39167a1" (UID: "7016d668-cd4b-4860-9908-2a97e39167a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:24.067935 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.067863    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs" pod "7016d668-cd4b-4860-9908-2a97e39167a1" (UID: "7016d668-cd4b-4860-9908-2a97e39167a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:24.163967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.163921    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:24.163967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.163956    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7016d668-cd4b-4860-9908-2a97e39167a1-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:24.163967 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.163969    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9jcr\" (UniqueName: \"kubernetes.io/projected/7016d668-cd4b-4860-9908-2a97e39167a1-kube-api-access-x9jcr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:24.445148 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.445107    4476 generic.go:356] "Generic (PLEG): container finished" podID="7016d668-cd4b-4860-9908-2a97e39167a1" containerID="2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02" exitCode=0
Nov 02 18:50:24.445359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.445162    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"
Nov 02 18:50:24.445359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.445194    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" event={"ID":"7016d668-cd4b-4860-9908-2a97e39167a1","Type":"ContainerDied","Data":"2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02"}
Nov 02 18:50:24.445359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.445245    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx" event={"ID":"7016d668-cd4b-4860-9908-2a97e39167a1","Type":"ContainerDied","Data":"246d832ca321962011796cf88ea6067362faf07016ffbe0e9cf68ad807498d15"}
Nov 02 18:50:24.445359 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.445272    4476 scope.go:122] "RemoveContainer" containerID="2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02"
Nov 02 18:50:24.456663 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.456616    4476 scope.go:122] "RemoveContainer" containerID="2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02"
Nov 02 18:50:24.456956 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:24.456934    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02\": container with ID starting with 2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02 not found: ID does not exist" containerID="2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02"
Nov 02 18:50:24.457016 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.456966    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02"} err="failed to get container status \"2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02\": rpc error: code = NotFound desc = could not find container \"2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02\": container with ID starting with 2f986983c81f08d1b7aa9a5c274fecb6935b394192e210e690d83a959b133b02 not found: ID does not exist"
Nov 02 18:50:24.610619 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.610563    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"]
Nov 02 18:50:24.613698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:24.613671    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-dc5689-5858b7c864-gmwwx"]
Nov 02 18:50:25.383560 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.383513    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7016d668-cd4b-4860-9908-2a97e39167a1" path="/var/lib/kubelet/pods/7016d668-cd4b-4860-9908-2a97e39167a1/volumes"
Nov 02 18:50:25.779104 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779000    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"]
Nov 02 18:50:25.779473 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779453    4476 cpu_manager.go:414] "containerMap: removing container" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779476    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779491    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" containerName="maas-api"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779496    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" containerName="maas-api"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779504    4476 cpu_manager.go:414] "containerMap: removing container" podUID="44f022ae-9bf4-4ecd-a105-08b66edf63ee" containerName="main"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779509    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f022ae-9bf4-4ecd-a105-08b66edf63ee" containerName="main"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779516    4476 cpu_manager.go:414] "containerMap: removing container" podUID="001c983b-7313-493d-9fec-04b983993105" containerName="payload-pre-processing"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779521    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001c983b-7313-493d-9fec-04b983993105" containerName="payload-pre-processing"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779531    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f0018d34-408d-4cf5-8853-6b08d369fbbb" containerName="main"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779537    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0018d34-408d-4cf5-8853-6b08d369fbbb" containerName="main"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779548    4476 cpu_manager.go:414] "containerMap: removing container" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779555    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779571    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779579    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e372001-8a06-47f1-ba3b-17ee0f102102" containerName="cleanup"
Nov 02 18:50:25.779587 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779593    4476 cpu_manager.go:414] "containerMap: removing container" podUID="60d64f4b-7a32-4013-96dd-415fcd0d5e2f" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779599    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d64f4b-7a32-4013-96dd-415fcd0d5e2f" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779605    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d2f06102-0149-4901-aaa5-e13243f51838" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779609    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f06102-0149-4901-aaa5-e13243f51838" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779637    4476 cpu_manager.go:414] "containerMap: removing container" podUID="8d97ceb9-8398-4bd7-aea8-174135bcd543" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779645    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d97ceb9-8398-4bd7-aea8-174135bcd543" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779655    4476 cpu_manager.go:414] "containerMap: removing container" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779660    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80674c25-365d-4314-8b60-8e393f630983" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779667    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f54391c3-dd1f-4e0f-b407-e91611f79e93" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779671    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54391c3-dd1f-4e0f-b407-e91611f79e93" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779678    4476 cpu_manager.go:414] "containerMap: removing container" podUID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779683    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779697    4476 cpu_manager.go:414] "containerMap: removing container" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779703    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779715    4476 cpu_manager.go:414] "containerMap: removing container" podUID="3fcae5a9-0fdc-4c39-8025-e5f714fbf883" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779720    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fcae5a9-0fdc-4c39-8025-e5f714fbf883" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779727    4476 cpu_manager.go:414] "containerMap: removing container" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779731    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779737    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ef20ad96-c209-4313-8a19-02c3d9b5d999" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779742    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef20ad96-c209-4313-8a19-02c3d9b5d999" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779748    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0b26c2bf-2217-4f94-9005-b739811a8667" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779753    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b26c2bf-2217-4f94-9005-b739811a8667" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779760    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fdea7084-f2eb-4883-ae0e-132fd5041b96" containerName="revoke-keys"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779764    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdea7084-f2eb-4883-ae0e-132fd5041b96" containerName="revoke-keys"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779772    4476 cpu_manager.go:414] "containerMap: removing container" podUID="642a337b-9bbb-4202-bc14-57bfa7869568" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779776    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642a337b-9bbb-4202-bc14-57bfa7869568" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779784    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779789    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779793    4476 cpu_manager.go:414] "containerMap: removing container" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779799    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779805    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4b823f43-2b94-4f45-a604-fed01945ae56" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779809    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b823f43-2b94-4f45-a604-fed01945ae56" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779818    4476 cpu_manager.go:414] "containerMap: removing container" podUID="fab4a298-dba5-4162-b226-e89896848f14" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779822    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab4a298-dba5-4162-b226-e89896848f14" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779831    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7016d668-cd4b-4860-9908-2a97e39167a1" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779835    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7016d668-cd4b-4860-9908-2a97e39167a1" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779840    4476 cpu_manager.go:414] "containerMap: removing container" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779847    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779856    4476 cpu_manager.go:414] "containerMap: removing container" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779863    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779872    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779880    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779973    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fcae5a9-0fdc-4c39-8025-e5f714fbf883" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779983    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c22053b-aceb-4f84-a364-78c65b651ac8" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779989    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef20ad96-c209-4313-8a19-02c3d9b5d999" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.779994    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b26c2bf-2217-4f94-9005-b739811a8667" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780000    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="71919475-10bb-4d20-a936-30e2a871b69f" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780005    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdea7084-f2eb-4883-ae0e-132fd5041b96" containerName="revoke-keys"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780010    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="642a337b-9bbb-4202-bc14-57bfa7869568" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780017    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780025    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a85290-1c16-4c6b-8cfe-89922ebac737" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780030    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b823f43-2b94-4f45-a604-fed01945ae56" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780036    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab4a298-dba5-4162-b226-e89896848f14" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780041    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7016d668-cd4b-4860-9908-2a97e39167a1" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780046    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80b6429-fd80-4c59-a1a6-b0b991667f03" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780051    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc33efd-3688-4ef5-8298-00f8d0e7554b" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780056    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e24920-a5e4-46a2-b782-3eb5d22f3b1e" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780061    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ad59c1-566a-4f0f-99d4-5e3df926ca5a" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780066    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f022ae-9bf4-4ecd-a105-08b66edf63ee" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780071    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="001c983b-7313-493d-9fec-04b983993105" containerName="payload-pre-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780078    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0018d34-408d-4cf5-8853-6b08d369fbbb" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780083    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9e95d7-9f83-4388-b59b-7e14307e88ba" containerName="istio-proxy"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780090    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d58be1-880c-4fc2-8104-ddc4d02a3adf" containerName="cleanup"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780096    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d64f4b-7a32-4013-96dd-415fcd0d5e2f" containerName="maas-api"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780102    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f06102-0149-4901-aaa5-e13243f51838" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780106    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d97ceb9-8398-4bd7-aea8-174135bcd543" containerName="main"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780111    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54391c3-dd1f-4e0f-b407-e91611f79e93" containerName="payload-processing"
Nov 02 18:50:25.780419 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.780117    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f24edb2-2853-4d83-ad30-4d4d64d7823c" containerName="main"
Nov 02 18:50:25.784439 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.784412    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.894254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.891200    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"]
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.901872    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmdl\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.901941    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.901978    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.902012    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.902061    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.902086    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.902125    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.902151    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.902654 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.902278    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:25.924754 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:25.924722    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\""
Nov 02 18:50:26.003970 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.003904    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004004    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmdl\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004053    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004454 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004423    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004450    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004530 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004484    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004539    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004567    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004608    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.004698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004656    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.005001 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.004952    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.005173 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.005148    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.005348 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.005320    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.005432 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.005311    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.007169 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.007148    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.007548 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.007512    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.062702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.062029    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.062702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.062145    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmdl\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.099870 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.099825    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:26.547324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.547296    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"]
Nov 02 18:50:26.548750 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:50:26.548723    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd319b8cb_0373_4ae3_8da5_bb2ceeba31b0.slice/crio-8753248358c28cc793f67a85df417deb801a7d827ff39c154d900a24941e9e97 WatchSource:0}: Error finding container 8753248358c28cc793f67a85df417deb801a7d827ff39c154d900a24941e9e97: Status 404 returned error can't find the container with id 8753248358c28cc793f67a85df417deb801a7d827ff39c154d900a24941e9e97
Nov 02 18:50:26.551010 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.550978    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:50:26.551123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.551043    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:50:26.551123 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:26.551070    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:50:27.478046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:27.478009    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" event={"ID":"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0","Type":"ContainerStarted","Data":"0158701d3dd13f8c141b7b28c85f00666ba42b52640234c52782ff8b9e994b9d"}
Nov 02 18:50:27.478270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:27.478069    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" event={"ID":"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0","Type":"ContainerStarted","Data":"8753248358c28cc793f67a85df417deb801a7d827ff39c154d900a24941e9e97"}
Nov 02 18:50:27.664655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:27.664195    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" podStartSLOduration=2.664173699 podStartE2EDuration="2.664173699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:27.663822641 +0000 UTC m=+5258.504428590" watchObservedRunningTime="2026-11-02 18:50:27.664173699 +0000 UTC m=+5258.504779633"
Nov 02 18:50:28.100388 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:28.100344    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:28.102354 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:28.102321    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 02 18:50:28.102490 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:28.102367    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 02 18:50:28.930189 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:28.930156    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"]
Nov 02 18:50:29.099786 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:29.099740    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:50:29.101198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:29.101175    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 02 18:50:29.101276 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:29.101222    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 02 18:50:29.492208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:29.492172    4476 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" secret="" err="secret \"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\" not found"
Nov 02 18:50:30.101115 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:30.101076    4476 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 02 18:50:30.101509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:30.101135    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 02 18:50:30.498581 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:30.498478    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy" containerID="cri-o://0158701d3dd13f8c141b7b28c85f00666ba42b52640234c52782ff8b9e994b9d" gracePeriod=30
Nov 02 18:50:31.062162 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:31.062126    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"]
Nov 02 18:50:36.063254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:36.063208    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerName="payload-processing" containerID="cri-o://711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839" gracePeriod=25
Nov 02 18:50:37.817230 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:37.817193    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:38.789108 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.789060    4476 generic.go:356] "Generic (PLEG): container finished" podID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerID="0158701d3dd13f8c141b7b28c85f00666ba42b52640234c52782ff8b9e994b9d" exitCode=0
Nov 02 18:50:38.789270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.789114    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" event={"ID":"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0","Type":"ContainerDied","Data":"0158701d3dd13f8c141b7b28c85f00666ba42b52640234c52782ff8b9e994b9d"}
Nov 02 18:50:38.854977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.854950    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:38.935991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.935896    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.935991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.935944    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.935991 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.935972    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.935997    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936036    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936118    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936169    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936194    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936210    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:38.936310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936225    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs\") pod \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\" (UID: \"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0\") "
Nov 02 18:50:38.936714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936446    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:38.936714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936581    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:38.936714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936600    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:38.936714 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936697    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:38.936977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936712    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:38.936977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.936733    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:38.938599 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.938569    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "kube-api-access-sdmdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:38.938742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.938662    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:38.938787 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.938763    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:50:38.938821 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:38.938775    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy" pod "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" (UID: "d319b8cb-0373-4ae3-8da5-bb2ceeba31b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:39.038270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038233    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmdl\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-kube-api-access-sdmdl\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.038270 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038272    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.038509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038288    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.038509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038304    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.038509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038317    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.038509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038331    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.038509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.038343    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:39.797836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.797758    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"
Nov 02 18:50:39.797836 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.797801    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6" event={"ID":"d319b8cb-0373-4ae3-8da5-bb2ceeba31b0","Type":"ContainerDied","Data":"8753248358c28cc793f67a85df417deb801a7d827ff39c154d900a24941e9e97"}
Nov 02 18:50:39.798059 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:39.797874    4476 scope.go:122] "RemoveContainer" containerID="0158701d3dd13f8c141b7b28c85f00666ba42b52640234c52782ff8b9e994b9d"
Nov 02 18:50:40.011750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:40.011704    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"]
Nov 02 18:50:40.059884 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:40.059800    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-626f6"]
Nov 02 18:50:41.382297 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:41.382263    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" path="/var/lib/kubelet/pods/d319b8cb-0373-4ae3-8da5-bb2ceeba31b0/volumes"
Nov 02 18:50:42.260254 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:42.260218    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"]
Nov 02 18:50:42.260533 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:42.260419    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" containerID="cri-o://e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6" gracePeriod=30
Nov 02 18:50:44.956416 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:44.956380    4476 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:50:44.956849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:44.956437    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:50:47.526141 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.526110    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:50:47.619051 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619014    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619088    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619142    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619190    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619226 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619217    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619240    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619281    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:47.619463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619302    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619404    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619477    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo\") pod \"e04e3f0d-9dae-4657-ab2e-de705905edf7\" (UID: \"e04e3f0d-9dae-4657-ab2e-de705905edf7\") "
Nov 02 18:50:47.619738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619557    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:47.619738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619556    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:47.619738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619623    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:47.619738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619683    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:47.620178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619910    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.620178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619939    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.620178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619950    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.620178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619959    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04e3f0d-9dae-4657-ab2e-de705905edf7-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.620178 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.619967    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.621658 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.621610    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:50:47.621803 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.621788    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:50:47.622137 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.622120    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "kube-api-access-khccr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:47.622192 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.622137    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token" pod "e04e3f0d-9dae-4657-ab2e-de705905edf7" (UID: "e04e3f0d-9dae-4657-ab2e-de705905edf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:47.721187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.721153    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khccr\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-kube-api-access-khccr\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.721187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.721181    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.721187 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.721191    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.721426 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.721199    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04e3f0d-9dae-4657-ab2e-de705905edf7-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:47.816418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.816380    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:50:47.855966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.855935    4476 generic.go:356] "Generic (PLEG): container finished" podID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerID="e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6" exitCode=0
Nov 02 18:50:47.856145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.855980    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"
Nov 02 18:50:47.856145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.856004    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" event={"ID":"e04e3f0d-9dae-4657-ab2e-de705905edf7","Type":"ContainerDied","Data":"e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6"}
Nov 02 18:50:47.856145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.856045    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw" event={"ID":"e04e3f0d-9dae-4657-ab2e-de705905edf7","Type":"ContainerDied","Data":"95fa01db789a93f175f3c4e715fb38f61cd37bfdfd545abba26ae065c2656931"}
Nov 02 18:50:47.856145 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.856066    4476 scope.go:122] "RemoveContainer" containerID="e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6"
Nov 02 18:50:47.866791 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.866765    4476 scope.go:122] "RemoveContainer" containerID="e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6"
Nov 02 18:50:47.867065 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:47.867048    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6\": container with ID starting with e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6 not found: ID does not exist" containerID="e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6"
Nov 02 18:50:47.867129 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:47.867072    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6"} err="failed to get container status \"e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6\": rpc error: code = NotFound desc = could not find container \"e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6\": container with ID starting with e16b1a8e337ba85f5cf1eb6ad0657389b132311941dc011e26450d77951470d6 not found: ID does not exist"
Nov 02 18:50:48.019338 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:48.019241    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"]
Nov 02 18:50:48.022411 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:48.022389    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-qt5pw"]
Nov 02 18:50:49.384833 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:49.384800    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" path="/var/lib/kubelet/pods/e04e3f0d-9dae-4657-ab2e-de705905edf7/volumes"
Nov 02 18:50:56.322459 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.322400    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 18:50:56.403512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.403483    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume\") pod \"844410f3-6b62-40c3-af5d-3dfcda2c7977\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") "
Nov 02 18:50:56.403705 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.403539    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs\" (UniqueName: \"kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs\") pod \"844410f3-6b62-40c3-af5d-3dfcda2c7977\" (UID: \"844410f3-6b62-40c3-af5d-3dfcda2c7977\") "
Nov 02 18:50:56.403980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.403959    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume" pod "844410f3-6b62-40c3-af5d-3dfcda2c7977" (UID: "844410f3-6b62-40c3-af5d-3dfcda2c7977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:56.405858 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.405835    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs" pod "844410f3-6b62-40c3-af5d-3dfcda2c7977" (UID: "844410f3-6b62-40c3-af5d-3dfcda2c7977"). InnerVolumeSpecName "kube-api-access-zpjvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:56.505347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.505316    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844410f3-6b62-40c3-af5d-3dfcda2c7977-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:56.505347 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.505342    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjvs\" (UniqueName: \"kubernetes.io/projected/844410f3-6b62-40c3-af5d-3dfcda2c7977-kube-api-access-zpjvs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:50:56.917026 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.916992    4476 generic.go:356] "Generic (PLEG): container finished" podID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerID="711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839" exitCode=0
Nov 02 18:50:56.917223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.917036    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"
Nov 02 18:50:56.917223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.917077    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" event={"ID":"844410f3-6b62-40c3-af5d-3dfcda2c7977","Type":"ContainerDied","Data":"711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839"}
Nov 02 18:50:56.917223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.917134    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm" event={"ID":"844410f3-6b62-40c3-af5d-3dfcda2c7977","Type":"ContainerDied","Data":"26a399c86bef4463615f1974eac0eaf85c8178a46ee731b8256f95159e20d648"}
Nov 02 18:50:56.917223 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.917156    4476 scope.go:122] "RemoveContainer" containerID="711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839"
Nov 02 18:50:56.938418 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.938395    4476 scope.go:122] "RemoveContainer" containerID="711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839"
Nov 02 18:50:56.938736 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:56.938715    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839\": container with ID starting with 711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839 not found: ID does not exist" containerID="711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839"
Nov 02 18:50:56.938798 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:56.938746    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839"} err="failed to get container status \"711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839\": rpc error: code = NotFound desc = could not find container \"711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839\": container with ID starting with 711f6a0a190208057be035305403b8cbf6b465346e8273da9917daaba5669839 not found: ID does not exist"
Nov 02 18:50:57.088415 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:57.088384    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"]
Nov 02 18:50:57.137672 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:57.137639    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a099c3-7c4699d489-gh5zm"]
Nov 02 18:50:57.382550 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:50:57.382515    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" path="/var/lib/kubelet/pods/844410f3-6b62-40c3-af5d-3dfcda2c7977/volumes"
Nov 02 18:50:59.058880 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:50:59.058843    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:51:29.059497 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:51:29.059452    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:51:29.103739 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:29.103705    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"]
Nov 02 18:51:34.104350 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:34.104310    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" containerName="payload-pre-processing" containerID="cri-o://3bff335b0642215f9ac7e8b30fb324bfefd826d784f2b94bdd89bf15e91aa503" gracePeriod=25
Nov 02 18:51:35.071728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:35.071678    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:51:45.072364 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:45.072312    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:51:54.249891 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.249857    4476 generic.go:356] "Generic (PLEG): container finished" podID="f3473e48-f176-45c0-a801-da43ccf16d65" containerID="3bff335b0642215f9ac7e8b30fb324bfefd826d784f2b94bdd89bf15e91aa503" exitCode=0
Nov 02 18:51:54.250398 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.249934    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" event={"ID":"f3473e48-f176-45c0-a801-da43ccf16d65","Type":"ContainerDied","Data":"3bff335b0642215f9ac7e8b30fb324bfefd826d784f2b94bdd89bf15e91aa503"}
Nov 02 18:51:54.352829 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.352804    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:51:54.454987 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.454903    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt\" (UniqueName: \"kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt\") pod \"f3473e48-f176-45c0-a801-da43ccf16d65\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") "
Nov 02 18:51:54.454987 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.454971    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume\") pod \"f3473e48-f176-45c0-a801-da43ccf16d65\" (UID: \"f3473e48-f176-45c0-a801-da43ccf16d65\") "
Nov 02 18:51:54.455429 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.455407    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume" pod "f3473e48-f176-45c0-a801-da43ccf16d65" (UID: "f3473e48-f176-45c0-a801-da43ccf16d65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:51:54.457339 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.457313    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt" pod "f3473e48-f176-45c0-a801-da43ccf16d65" (UID: "f3473e48-f176-45c0-a801-da43ccf16d65"). InnerVolumeSpecName "kube-api-access-chxxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:54.556869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.556838    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxxt\" (UniqueName: \"kubernetes.io/projected/f3473e48-f176-45c0-a801-da43ccf16d65-kube-api-access-chxxt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:54.556869 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:54.556868    4476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3473e48-f176-45c0-a801-da43ccf16d65-plugins-config-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:51:55.259538 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:55.259496    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52" event={"ID":"f3473e48-f176-45c0-a801-da43ccf16d65","Type":"ContainerDied","Data":"fffdf9ed753cd133fb97bb19fdf1c25609664abb99d18d3265d822accd5fa6fd"}
Nov 02 18:51:55.259538 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:55.259540    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"
Nov 02 18:51:55.260119 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:55.259574    4476 scope.go:122] "RemoveContainer" containerID="3bff335b0642215f9ac7e8b30fb324bfefd826d784f2b94bdd89bf15e91aa503"
Nov 02 18:51:55.422644 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:55.422599    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"]
Nov 02 18:51:55.472409 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:55.472374    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f7aa2e-5884c47999-kmv52"]
Nov 02 18:51:57.382080 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:57.382046    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" path="/var/lib/kubelet/pods/f3473e48-f176-45c0-a801-da43ccf16d65/volumes"
Nov 02 18:51:58.044697 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:58.044663    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"]
Nov 02 18:51:58.094403 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:58.094364    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545aabe6-c4kkb"]
Nov 02 18:51:59.058751 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:51:59.058715    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:51:59.391522 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:51:59.391480    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdea7084-f2eb-4883-ae0e-132fd5041b96" path="/var/lib/kubelet/pods/fdea7084-f2eb-4883-ae0e-132fd5041b96/volumes"
Nov 02 18:52:20.090485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:20.090451    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"]
Nov 02 18:52:20.090981 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:20.090685    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy" containerID="cri-o://542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222" gracePeriod=30
Nov 02 18:52:25.348997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.348969    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:52:25.433142 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.433110    4476 generic.go:356] "Generic (PLEG): container finished" podID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerID="542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222" exitCode=0
Nov 02 18:52:25.433324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.433161    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"
Nov 02 18:52:25.433324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.433190    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" event={"ID":"f5269bdd-c304-425e-ae66-f6d1b2242bb3","Type":"ContainerDied","Data":"542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222"}
Nov 02 18:52:25.433324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.433247    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq" event={"ID":"f5269bdd-c304-425e-ae66-f6d1b2242bb3","Type":"ContainerDied","Data":"e0b93509a6860e50daa3bdf564875324f9f25519a77938bb4e8cfc9dc8c8a87e"}
Nov 02 18:52:25.433324 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.433270    4476 scope.go:122] "RemoveContainer" containerID="542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222"
Nov 02 18:52:25.442980 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.442949    4476 scope.go:122] "RemoveContainer" containerID="39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"
Nov 02 18:52:25.452018 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.451997    4476 scope.go:122] "RemoveContainer" containerID="542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222"
Nov 02 18:52:25.452329 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:25.452309    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222\": container with ID starting with 542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222 not found: ID does not exist" containerID="542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222"
Nov 02 18:52:25.452424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452343    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222"} err="failed to get container status \"542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222\": rpc error: code = NotFound desc = could not find container \"542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222\": container with ID starting with 542371a93ca912cfe3f61c662b7c791da5027f48963438a185eea9cdbf2b8222 not found: ID does not exist"
Nov 02 18:52:25.452424 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452363    4476 scope.go:122] "RemoveContainer" containerID="39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"
Nov 02 18:52:25.452551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452423    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452476    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452551 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452511    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452549    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452734 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452696    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452737    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:25.452845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452753    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452792    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.452845 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:25.452825    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67\": container with ID starting with 39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67 not found: ID does not exist" containerID="39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"
Nov 02 18:52:25.452845 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452841    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.453106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452853    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67"} err="failed to get container status \"39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67\": rpc error: code = NotFound desc = could not find container \"39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67\": container with ID starting with 39b6d068d4c031a05cb6987f5933ed0dcef8453cbcfdba7ef2155b03fbfe5a67 not found: ID does not exist"
Nov 02 18:52:25.453106 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.452888    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data\") pod \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\" (UID: \"f5269bdd-c304-425e-ae66-f6d1b2242bb3\") "
Nov 02 18:52:25.454462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.453539    4476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.454462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.453967    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:25.454462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.454248    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:25.454462 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.454310    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:25.454848 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.454549    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:25.455186 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.455164    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:25.455381 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.455363    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:25.456188 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.456161    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "kube-api-access-cnb6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:25.456568 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.456546    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo" pod "f5269bdd-c304-425e-ae66-f6d1b2242bb3" (UID: "f5269bdd-c304-425e-ae66-f6d1b2242bb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:52:25.498379 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.498348    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"]
Nov 02 18:52:25.507197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.507170    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"]
Nov 02 18:52:25.515100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.515074    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"]
Nov 02 18:52:25.554149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554118    4476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-token\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554149 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554151    4476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-credential-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554167    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnb6d\" (UniqueName: \"kubernetes.io/projected/f5269bdd-c304-425e-ae66-f6d1b2242bb3-kube-api-access-cnb6d\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554184    4476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-workload-socket\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554197    4476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-data\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554212    4476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-envoy\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554225    4476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istio-podinfo\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:25.554404 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:25.554238    4476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5269bdd-c304-425e-ae66-f6d1b2242bb3-istiod-ca-cert\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:26.050162 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:26.050073    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"]
Nov 02 18:52:26.053695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:26.053664    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2a1bcc-openshift-default-54bc7bf659-qjfpq"]
Nov 02 18:52:27.382357 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:27.382322    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" path="/var/lib/kubelet/pods/f5269bdd-c304-425e-ae66-f6d1b2242bb3/volumes"
Nov 02 18:52:29.058236 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:29.058198    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:52:40.519401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.519325    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerName="main" containerID="cri-o://01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864" gracePeriod=45
Nov 02 18:52:40.524535 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.524504    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": EOF"
Nov 02 18:52:40.528032 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.528005    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" podUID="eb1731d5-2413-48d5-932a-cf777619c149" containerName="main" containerID="cri-o://5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe" gracePeriod=45
Nov 02 18:52:40.535114 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.535090    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerName="main" containerID="cri-o://180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688" gracePeriod=45
Nov 02 18:52:40.660264 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.660217    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" podUID="eb1731d5-2413-48d5-932a-cf777619c149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused"
Nov 02 18:52:40.845047 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.844585    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 18:52:40.858105 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.858077    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 18:52:40.865003 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:40.864983    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 18:52:41.010485 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010445    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm\") pod \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") "
Nov 02 18:52:41.010707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010502    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs\") pod \"eb1731d5-2413-48d5-932a-cf777619c149\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") "
Nov 02 18:52:41.010707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010545    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home\") pod \"eb1731d5-2413-48d5-932a-cf777619c149\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") "
Nov 02 18:52:41.010707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010571    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs\") pod \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") "
Nov 02 18:52:41.010707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010648    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache\") pod \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") "
Nov 02 18:52:41.010707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010675    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir\") pod \"eb1731d5-2413-48d5-932a-cf777619c149\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") "
Nov 02 18:52:41.010707 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010697    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm\") pod \"eb1731d5-2413-48d5-932a-cf777619c149\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010725    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx\" (UniqueName: \"kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx\") pod \"eb1731d5-2413-48d5-932a-cf777619c149\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010745    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home\") pod \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010769    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir\") pod \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010806    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir\") pod \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010842    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache\") pod \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010876    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d\" (UniqueName: \"kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d\") pod \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010890    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home" pod "eb1731d5-2413-48d5-932a-cf777619c149" (UID: "eb1731d5-2413-48d5-932a-cf777619c149"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010935    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home\") pod \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.010964    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm\") pod \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.011015    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667\" (UniqueName: \"kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667\") pod \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\" (UID: \"f8ffa922-9e25-4195-82d3-d7c9f438f65e\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.011049    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache\") pod \"eb1731d5-2413-48d5-932a-cf777619c149\" (UID: \"eb1731d5-2413-48d5-932a-cf777619c149\") "
Nov 02 18:52:41.011107 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.011106    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs\") pod \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\" (UID: \"f7a8d9e2-17ee-4b30-9240-3c33043cc156\") "
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.011413    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir" pod "f8ffa922-9e25-4195-82d3-d7c9f438f65e" (UID: "f8ffa922-9e25-4195-82d3-d7c9f438f65e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.012220    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache" pod "f8ffa922-9e25-4195-82d3-d7c9f438f65e" (UID: "f8ffa922-9e25-4195-82d3-d7c9f438f65e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.012214    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache" pod "f7a8d9e2-17ee-4b30-9240-3c33043cc156" (UID: "f7a8d9e2-17ee-4b30-9240-3c33043cc156"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.012742    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home" pod "f7a8d9e2-17ee-4b30-9240-3c33043cc156" (UID: "f7a8d9e2-17ee-4b30-9240-3c33043cc156"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.012810    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home" pod "f8ffa922-9e25-4195-82d3-d7c9f438f65e" (UID: "f8ffa922-9e25-4195-82d3-d7c9f438f65e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.012821    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir" pod "f7a8d9e2-17ee-4b30-9240-3c33043cc156" (UID: "f7a8d9e2-17ee-4b30-9240-3c33043cc156"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.013117    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache" pod "eb1731d5-2413-48d5-932a-cf777619c149" (UID: "eb1731d5-2413-48d5-932a-cf777619c149"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.015118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.014199    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir" pod "eb1731d5-2413-48d5-932a-cf777619c149" (UID: "eb1731d5-2413-48d5-932a-cf777619c149"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.016283    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm" pod "f7a8d9e2-17ee-4b30-9240-3c33043cc156" (UID: "f7a8d9e2-17ee-4b30-9240-3c33043cc156"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017022    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm" pod "eb1731d5-2413-48d5-932a-cf777619c149" (UID: "eb1731d5-2413-48d5-932a-cf777619c149"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017102    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs" pod "eb1731d5-2413-48d5-932a-cf777619c149" (UID: "eb1731d5-2413-48d5-932a-cf777619c149"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017128    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d" pod "f7a8d9e2-17ee-4b30-9240-3c33043cc156" (UID: "f7a8d9e2-17ee-4b30-9240-3c33043cc156"). InnerVolumeSpecName "kube-api-access-mj57d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017151    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667" pod "f8ffa922-9e25-4195-82d3-d7c9f438f65e" (UID: "f8ffa922-9e25-4195-82d3-d7c9f438f65e"). InnerVolumeSpecName "kube-api-access-kv667". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017157    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs" pod "f7a8d9e2-17ee-4b30-9240-3c33043cc156" (UID: "f7a8d9e2-17ee-4b30-9240-3c33043cc156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017210    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs" pod "f8ffa922-9e25-4195-82d3-d7c9f438f65e" (UID: "f8ffa922-9e25-4195-82d3-d7c9f438f65e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.017454    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx" pod "eb1731d5-2413-48d5-932a-cf777619c149" (UID: "eb1731d5-2413-48d5-932a-cf777619c149"). InnerVolumeSpecName "kube-api-access-gqwdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:41.020643 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.018580    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm" pod "f8ffa922-9e25-4195-82d3-d7c9f438f65e" (UID: "f8ffa922-9e25-4195-82d3-d7c9f438f65e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:52:41.112286 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112254    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112297    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj57d\" (UniqueName: \"kubernetes.io/projected/f7a8d9e2-17ee-4b30-9240-3c33043cc156-kube-api-access-mj57d\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112314    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112328    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112342    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv667\" (UniqueName: \"kubernetes.io/projected/f8ffa922-9e25-4195-82d3-d7c9f438f65e-kube-api-access-kv667\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112355    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112368    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112381    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112394    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb1731d5-2413-48d5-932a-cf777619c149-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112406    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112419    4476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tls-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112431    4476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-model-cache\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112444 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112444    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112457    4476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb1731d5-2413-48d5-932a-cf777619c149-dshm\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112471    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqwdx\" (UniqueName: \"kubernetes.io/projected/eb1731d5-2413-48d5-932a-cf777619c149-kube-api-access-gqwdx\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112484    4476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-home\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112496    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a8d9e2-17ee-4b30-9240-3c33043cc156-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.112861 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.112509    4476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ffa922-9e25-4195-82d3-d7c9f438f65e-tmp-dir\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:41.548725 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.548687    4476 generic.go:356] "Generic (PLEG): container finished" podID="eb1731d5-2413-48d5-932a-cf777619c149" containerID="5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe" exitCode=0
Nov 02 18:52:41.549287 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.548732    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" event={"ID":"eb1731d5-2413-48d5-932a-cf777619c149","Type":"ContainerDied","Data":"5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe"}
Nov 02 18:52:41.549463 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.549443    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk" event={"ID":"eb1731d5-2413-48d5-932a-cf777619c149","Type":"ContainerDied","Data":"8b7ea16310969345bf94efb373e914b57ee096682cc74a3799e103def20aadae"}
Nov 02 18:52:41.549583 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.548815    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"
Nov 02 18:52:41.549917 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.549471    4476 scope.go:122] "RemoveContainer" containerID="5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe"
Nov 02 18:52:41.551977 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.551955    4476 generic.go:356] "Generic (PLEG): container finished" podID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerID="180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688" exitCode=0
Nov 02 18:52:41.552232 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.552212    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" event={"ID":"f7a8d9e2-17ee-4b30-9240-3c33043cc156","Type":"ContainerDied","Data":"180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688"}
Nov 02 18:52:41.552372 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.552357    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg" event={"ID":"f7a8d9e2-17ee-4b30-9240-3c33043cc156","Type":"ContainerDied","Data":"0db6312f203910f4f4db27975346a7d178dc53cef209b5fd890d067900a3b7d4"}
Nov 02 18:52:41.552524 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.552510    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"
Nov 02 18:52:41.555933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.555767    4476 generic.go:356] "Generic (PLEG): container finished" podID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerID="01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864" exitCode=0
Nov 02 18:52:41.555933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.555825    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"
Nov 02 18:52:41.555933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.555854    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" event={"ID":"f8ffa922-9e25-4195-82d3-d7c9f438f65e","Type":"ContainerDied","Data":"01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864"}
Nov 02 18:52:41.555933 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.555883    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l" event={"ID":"f8ffa922-9e25-4195-82d3-d7c9f438f65e","Type":"ContainerDied","Data":"de1a9dd4eb71f392304755377cc8136199a7a971ba02537f9ad6ac43b4059661"}
Nov 02 18:52:41.563472 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.563432    4476 scope.go:122] "RemoveContainer" containerID="5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe"
Nov 02 18:52:41.563932 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:41.563897    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe\": container with ID starting with 5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe not found: ID does not exist" containerID="5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe"
Nov 02 18:52:41.564088 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.563942    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe"} err="failed to get container status \"5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe\": rpc error: code = NotFound desc = could not find container \"5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe\": container with ID starting with 5c283acf91db9bc560bf206c60632c6b785ac6c444bccc577a5d6c9c0730f7fe not found: ID does not exist"
Nov 02 18:52:41.564088 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.563966    4476 scope.go:122] "RemoveContainer" containerID="180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688"
Nov 02 18:52:41.575509 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.575477    4476 scope.go:122] "RemoveContainer" containerID="180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688"
Nov 02 18:52:41.575849 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:41.575824    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688\": container with ID starting with 180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688 not found: ID does not exist" containerID="180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688"
Nov 02 18:52:41.575983 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.575957    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688"} err="failed to get container status \"180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688\": rpc error: code = NotFound desc = could not find container \"180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688\": container with ID starting with 180d51faea03dc1b3e478b3261a672cbc0841397088f778c662c9453e78a5688 not found: ID does not exist"
Nov 02 18:52:41.576081 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.576071    4476 scope.go:122] "RemoveContainer" containerID="01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864"
Nov 02 18:52:41.585526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.585507    4476 scope.go:122] "RemoveContainer" containerID="01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864"
Nov 02 18:52:41.585827 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:41.585807    4476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864\": container with ID starting with 01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864 not found: ID does not exist" containerID="01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864"
Nov 02 18:52:41.585908 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.585835    4476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864"} err="failed to get container status \"01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864\": rpc error: code = NotFound desc = could not find container \"01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864\": container with ID starting with 01b24e4cac4d33b5c6cc5777393873f0c565778c1131cfc1ceef2ebb6c246864 not found: ID does not exist"
Nov 02 18:52:41.717664 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.717557    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"]
Nov 02 18:52:41.721266 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.721242    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/simulator-w3-6e3175-kserve-65d6d9d6d5-v4dfg"]
Nov 02 18:52:41.828674 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.828610    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"]
Nov 02 18:52:41.831012 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.830986    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/premium-w3-6e3175-kserve-7f5db7b6d8-9bvwk"]
Nov 02 18:52:41.938932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.938889    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"]
Nov 02 18:52:41.941964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:41.941935    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-w3-6e3175-kserve-599596fc95-dg72l"]
Nov 02 18:52:43.381544 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:43.381508    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1731d5-2413-48d5-932a-cf777619c149" path="/var/lib/kubelet/pods/eb1731d5-2413-48d5-932a-cf777619c149/volumes"
Nov 02 18:52:43.381982 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:43.381928    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" path="/var/lib/kubelet/pods/f7a8d9e2-17ee-4b30-9240-3c33043cc156/volumes"
Nov 02 18:52:43.382272 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:43.382258    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" path="/var/lib/kubelet/pods/f8ffa922-9e25-4195-82d3-d7c9f438f65e/volumes"
Nov 02 18:52:46.341966 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.341919    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"]
Nov 02 18:52:46.342420 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.342183    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" podUID="13a1e239-aa47-4ff3-be58-6dba25de415a" containerName="maas-api" containerID="cri-o://df6134b879d78a7651ef4cb352abca6094bf54c6f231e9c026b5f32f09ad50aa" gracePeriod=30
Nov 02 18:52:46.592071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.592000    4476 generic.go:356] "Generic (PLEG): container finished" podID="13a1e239-aa47-4ff3-be58-6dba25de415a" containerID="df6134b879d78a7651ef4cb352abca6094bf54c6f231e9c026b5f32f09ad50aa" exitCode=0
Nov 02 18:52:46.592071 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.592061    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" event={"ID":"13a1e239-aa47-4ff3-be58-6dba25de415a","Type":"ContainerDied","Data":"df6134b879d78a7651ef4cb352abca6094bf54c6f231e9c026b5f32f09ad50aa"}
Nov 02 18:52:46.592224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.592108    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx" event={"ID":"13a1e239-aa47-4ff3-be58-6dba25de415a","Type":"ContainerDied","Data":"365758f45e99d055e39cf824c990b89ba902b759f6daf9c7c502aef63315fb82"}
Nov 02 18:52:46.592224 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.592124    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="365758f45e99d055e39cf824c990b89ba902b759f6daf9c7c502aef63315fb82"
Nov 02 18:52:46.604491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.604469    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 18:52:46.672702 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.672673    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs\") pod \"13a1e239-aa47-4ff3-be58-6dba25de415a\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") "
Nov 02 18:52:46.672891 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.672735    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9\" (UniqueName: \"kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9\") pod \"13a1e239-aa47-4ff3-be58-6dba25de415a\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") "
Nov 02 18:52:46.672891 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.672804    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\") pod \"13a1e239-aa47-4ff3-be58-6dba25de415a\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") "
Nov 02 18:52:46.672891 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.672873    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca\") pod \"13a1e239-aa47-4ff3-be58-6dba25de415a\" (UID: \"13a1e239-aa47-4ff3-be58-6dba25de415a\") "
Nov 02 18:52:46.673698 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.673665    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca" pod "13a1e239-aa47-4ff3-be58-6dba25de415a" (UID: "13a1e239-aa47-4ff3-be58-6dba25de415a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:46.675097 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.675074    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs" pod "13a1e239-aa47-4ff3-be58-6dba25de415a" (UID: "13a1e239-aa47-4ff3-be58-6dba25de415a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:46.675210 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.675113    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9" pod "13a1e239-aa47-4ff3-be58-6dba25de415a" (UID: "13a1e239-aa47-4ff3-be58-6dba25de415a"). InnerVolumeSpecName "kube-api-access-z9cz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:46.675480 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.675462    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls" pod "13a1e239-aa47-4ff3-be58-6dba25de415a" (UID: "13a1e239-aa47-4ff3-be58-6dba25de415a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:46.774288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.774251    4476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-metrics-certs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:46.774288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.774283    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9cz9\" (UniqueName: \"kubernetes.io/projected/13a1e239-aa47-4ff3-be58-6dba25de415a-kube-api-access-z9cz9\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:46.774288 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.774293    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13a1e239-aa47-4ff3-be58-6dba25de415a-maas-api-tls\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:46.774546 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:46.774302    4476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1e239-aa47-4ff3-be58-6dba25de415a-openshift-service-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:52:47.597208 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:47.597166    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"
Nov 02 18:52:47.760731 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:47.760643    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"]
Nov 02 18:52:47.810085 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:47.810048    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6e3175-56956776c4-ntjwx"]
Nov 02 18:52:49.391056 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:49.391024    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a1e239-aa47-4ff3-be58-6dba25de415a" path="/var/lib/kubelet/pods/13a1e239-aa47-4ff3-be58-6dba25de415a/volumes"
Nov 02 18:52:50.313118 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:50.313085    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:52:50.323779 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:50.323754    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:52:51.915793 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:52:51.915761    4476 scope.go:122] "RemoveContainer" containerID="df6134b879d78a7651ef4cb352abca6094bf54c6f231e9c026b5f32f09ad50aa"
Nov 02 18:52:59.059042 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:52:59.059005    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:53:06.703057 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:06.703024    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-685c598f54-4cmqn_e027aa27-0b9b-4c21-8e23-54644e537f96/authorino/0.log"
Nov 02 18:53:07.082152 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:07.082065    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rjmsd_4c792d4f-f412-4cc5-8e8f-07c797c31015/manager/0.log"
Nov 02 18:53:07.330790 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:07.330756    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hw69v_c0d81e3f-d92e-4a06-adab-a62d740747a0/manager/0.log"
Nov 02 18:53:07.558964 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:07.558927    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wlcdx_f4f28bac-e0b7-41dc-b254-0a3bdd5f997c/kuadrant-console-plugin/0.log"
Nov 02 18:53:08.429555 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:08.429516    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wtg8g_f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6/manager/0.log"
Nov 02 18:53:09.089022 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:09.088989    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wg9xn_93dd06a4-ba2c-474a-8a39-c3e591131472/manager/0.log"
Nov 02 18:53:10.469974 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:10.469941    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ws6kb_13454cdd-8e2d-4c3c-92cf-0c6adc13355f/manager/0.log"
Nov 02 18:53:10.946737 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:10.946703    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zt8p9_dee2ee5b-9244-4eaf-8ed6-e72a261d01f7/manager/0.log"
Nov 02 18:53:11.609830 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:11.609807    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w47j_76341d23-9cf8-4a8b-becb-16a32e0c9775/server/0.log"
Nov 02 18:53:11.835770 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:11.835744    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f9fdr_56c7fe1e-8ea8-454c-9ce4-2585f83c60b6/manager/0.log"
Nov 02 18:53:29.058883 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:53:29.058849    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:53:51.960063 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:53:51.960027    4476 scope.go:122] "RemoveContainer" containerID="c6dd2413695581fdb83ee9959a5c5d5819697afdef3888a231a9582c98b9cffd"
Nov 02 18:53:59.058085 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:53:59.058046    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:54:29.058955 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:54:29.058913    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:54:59.057958 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:54:59.057919    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:55:29.067379 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:55:29.067337    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:55:53.471932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.471850    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"]
Nov 02 18:55:53.472588 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472564    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerName="main"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472592    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerName="main"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472610    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerName="main"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472619    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerName="main"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472658    4476 cpu_manager.go:414] "containerMap: removing container" podUID="eb1731d5-2413-48d5-932a-cf777619c149" containerName="main"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472665    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1731d5-2413-48d5-932a-cf777619c149" containerName="main"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472688    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy"
Nov 02 18:55:53.472695 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472695    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472708    4476 cpu_manager.go:414] "containerMap: removing container" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerName="payload-processing"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472716    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerName="payload-processing"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472728    4476 cpu_manager.go:414] "containerMap: removing container" podUID="13a1e239-aa47-4ff3-be58-6dba25de415a" containerName="maas-api"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472734    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a1e239-aa47-4ff3-be58-6dba25de415a" containerName="maas-api"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472747    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472754    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472764    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" containerName="payload-pre-processing"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472773    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" containerName="payload-pre-processing"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472784    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472791    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472818    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472824    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy"
Nov 02 18:55:53.472928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472922    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04e3f0d-9dae-4657-ab2e-de705905edf7" containerName="istio-proxy"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472940    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a8d9e2-17ee-4b30-9240-3c33043cc156" containerName="main"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472949    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ffa922-9e25-4195-82d3-d7c9f438f65e" containerName="main"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472957    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1731d5-2413-48d5-932a-cf777619c149" containerName="main"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472967    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472974    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="844410f3-6b62-40c3-af5d-3dfcda2c7977" containerName="payload-processing"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472982    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a1e239-aa47-4ff3-be58-6dba25de415a" containerName="maas-api"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472990    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d319b8cb-0373-4ae3-8da5-bb2ceeba31b0" containerName="istio-proxy"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.472997    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3473e48-f176-45c0-a801-da43ccf16d65" containerName="payload-pre-processing"
Nov 02 18:55:53.473374 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.473006    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5269bdd-c304-425e-ae66-f6d1b2242bb3" containerName="istio-proxy"
Nov 02 18:55:53.476529 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.476508    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.569976 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.568095    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvkb\" (UniqueName: \"kubernetes.io/projected/792eac8f-2598-4b49-94d1-6d0231efb074-kube-api-access-njvkb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570022    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570061    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570134    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570169    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792eac8f-2598-4b49-94d1-6d0231efb074-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570195    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792eac8f-2598-4b49-94d1-6d0231efb074-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570224    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570271    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.570503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.570301    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792eac8f-2598-4b49-94d1-6d0231efb074-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.671352 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.671305    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.672356 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.672332    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792eac8f-2598-4b49-94d1-6d0231efb074-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.673503 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.673482    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792eac8f-2598-4b49-94d1-6d0231efb074-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.673963 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.673942    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.674696 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.674676    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.675298 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.675279    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792eac8f-2598-4b49-94d1-6d0231efb074-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.710045    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvkb\" (UniqueName: \"kubernetes.io/projected/792eac8f-2598-4b49-94d1-6d0231efb074-kube-api-access-njvkb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.710126    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.710361    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.674360    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.709139    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792eac8f-2598-4b49-94d1-6d0231efb074-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.673414    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792eac8f-2598-4b49-94d1-6d0231efb074-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.675099    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.672245    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.711786    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.712854    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"]
Nov 02 18:55:53.715512 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.714986    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792eac8f-2598-4b49-94d1-6d0231efb074-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.732969 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.732898    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792eac8f-2598-4b49-94d1-6d0231efb074-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.776655 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.772523    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvkb\" (UniqueName: \"kubernetes.io/projected/792eac8f-2598-4b49-94d1-6d0231efb074-kube-api-access-njvkb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-thm4f\" (UID: \"792eac8f-2598-4b49-94d1-6d0231efb074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:53.791155 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:53.789891    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:54.297818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.297791    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"]
Nov 02 18:55:54.301738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.300146    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:55:54.301738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.300232    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:55:54.301738 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.300274    4476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 18:55:54.653919 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.653885    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" event={"ID":"792eac8f-2598-4b49-94d1-6d0231efb074","Type":"ContainerStarted","Data":"a5ade60ed30c584eabf3bf98fa9183777d88ef440e98e75b5fce6c0bc0c60cc1"}
Nov 02 18:55:54.654457 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.654435    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" event={"ID":"792eac8f-2598-4b49-94d1-6d0231efb074","Type":"ContainerStarted","Data":"3172518cd872b0dec370c7a2aea1a097ce4bcfdbd3f6c0c7f5ec6e9f691ce60d"}
Nov 02 18:55:54.775164 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.775086    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podStartSLOduration=1.775067087 podStartE2EDuration="1.775067087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:54.771736575 +0000 UTC m=+5585.612342511" watchObservedRunningTime="2026-11-02 18:55:54.775067087 +0000 UTC m=+5585.615673022"
Nov 02 18:55:54.794506 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:54.794477    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:55:55.792741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:55.792704    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:55:55.793166 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:55.792776    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:55:56.791450 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:56.791409    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:55:56.791659 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:56.791516    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:55:57.791291 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:57.791253    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:55:57.791750 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:57.791320    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:55:58.790753 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:58.790718    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:55:58.790947 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:58.790782    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:55:59.081537 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:55:59.081446    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:55:59.791075 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:59.791040    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:55:59.791285 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:55:59.791099    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:56:00.790978 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:00.790941    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:56:00.791395 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:00.791005    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:56:01.791186 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:01.791149    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:56:01.791618 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:01.791201    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Nov 02 18:56:03.832794 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:03.832753    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:56:03.833284 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:03.832826    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:56:04.875781 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:04.875725    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:56:04.876497 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:04.876452    4476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:56:05.526818 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:05.526776    4476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:56:05.527198 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:05.527172    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:56:05.528041 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:56:05.527986    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f"
Nov 02 18:56:29.058165 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:56:29.058129    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:56:59.061567 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:56:59.061526    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:57:29.060653 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:57:29.060577    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:57:50.352997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.352968    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:57:50.366046 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.366014    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 18:57:50.826741 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.826707    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"]
Nov 02 18:57:50.830099 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.830081    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.883368 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.883338    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:57:50.883507 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.883372    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:57:50.887772 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.887752    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.887930 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.887913    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kww8t\" (UniqueName: \"kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.888028 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.887946    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.932997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.932941    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:57:50.932997 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.932957    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 18:57:50.935437 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.935420    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"]
Nov 02 18:57:50.988928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.988895    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kww8t\" (UniqueName: \"kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.988928 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.988927    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.989136 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.989040    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.989537 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.989511    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:50.991456 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:50.991440    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:51.047902 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:51.047877    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kww8t\" (UniqueName: \"kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t\") pod \"maas-api-revoke-sub-64b4b2f6-c987k\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:51.143200 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:51.143171    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:51.515101 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:51.515034    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"]
Nov 02 18:57:51.517553 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:57:51.517522    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aca7966_d0ce_43cf_ae4a_8738531eb118.slice/crio-b138df71b3bc7eefdd35241bcdc6ae794be577f649b4f6d81898ab5a3397961c WatchSource:0}: Error finding container b138df71b3bc7eefdd35241bcdc6ae794be577f649b4f6d81898ab5a3397961c: Status 404 returned error can't find the container with id b138df71b3bc7eefdd35241bcdc6ae794be577f649b4f6d81898ab5a3397961c
Nov 02 18:57:52.407943 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:52.407907    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k" event={"ID":"2aca7966-d0ce-43cf-ae4a-8738531eb118","Type":"ContainerStarted","Data":"ad48ed76d8c6fa01fbd2db0389807ac938cc906dd72404ab238c8b1107b8d1d9"}
Nov 02 18:57:52.408126 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:52.407980    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k" event={"ID":"2aca7966-d0ce-43cf-ae4a-8738531eb118","Type":"ContainerStarted","Data":"b138df71b3bc7eefdd35241bcdc6ae794be577f649b4f6d81898ab5a3397961c"}
Nov 02 18:57:52.522505 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:52.522453    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k" podStartSLOduration=2.522438285 podStartE2EDuration="2.522438285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:57:52.520100691 +0000 UTC m=+5703.360706637" watchObservedRunningTime="2026-11-02 18:57:52.522438285 +0000 UTC m=+5703.363044217"
Nov 02 18:57:53.414651 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:53.414588    4476 generic.go:356] "Generic (PLEG): container finished" podID="2aca7966-d0ce-43cf-ae4a-8738531eb118" containerID="ad48ed76d8c6fa01fbd2db0389807ac938cc906dd72404ab238c8b1107b8d1d9" exitCode=0
Nov 02 18:57:53.414842 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:53.414670    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k" event={"ID":"2aca7966-d0ce-43cf-ae4a-8738531eb118","Type":"ContainerDied","Data":"ad48ed76d8c6fa01fbd2db0389807ac938cc906dd72404ab238c8b1107b8d1d9"}
Nov 02 18:57:54.559467 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.559443    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:54.621872 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.621847    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca\") pod \"2aca7966-d0ce-43cf-ae4a-8738531eb118\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") "
Nov 02 18:57:54.622037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.621883    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca\") pod \"2aca7966-d0ce-43cf-ae4a-8738531eb118\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") "
Nov 02 18:57:54.622037 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.621936    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t\" (UniqueName: \"kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t\") pod \"2aca7966-d0ce-43cf-ae4a-8738531eb118\" (UID: \"2aca7966-d0ce-43cf-ae4a-8738531eb118\") "
Nov 02 18:57:54.622311 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.622276    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca" pod "2aca7966-d0ce-43cf-ae4a-8738531eb118" (UID: "2aca7966-d0ce-43cf-ae4a-8738531eb118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:57:54.624057 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.624030    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca" pod "2aca7966-d0ce-43cf-ae4a-8738531eb118" (UID: "2aca7966-d0ce-43cf-ae4a-8738531eb118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:57:54.624158 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.624133    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t" pod "2aca7966-d0ce-43cf-ae4a-8738531eb118" (UID: "2aca7966-d0ce-43cf-ae4a-8738531eb118"). InnerVolumeSpecName "kube-api-access-kww8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:57:54.722728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.722656    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:57:54.722728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.722682    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aca7966-d0ce-43cf-ae4a-8738531eb118-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:57:54.722728 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:54.722697    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kww8t\" (UniqueName: \"kubernetes.io/projected/2aca7966-d0ce-43cf-ae4a-8738531eb118-kube-api-access-kww8t\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:57:55.426516 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:55.426488    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"
Nov 02 18:57:55.426694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:55.426520    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k" event={"ID":"2aca7966-d0ce-43cf-ae4a-8738531eb118","Type":"ContainerDied","Data":"b138df71b3bc7eefdd35241bcdc6ae794be577f649b4f6d81898ab5a3397961c"}
Nov 02 18:57:55.426694 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:57:55.426567    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b138df71b3bc7eefdd35241bcdc6ae794be577f649b4f6d81898ab5a3397961c"
Nov 02 18:57:59.057507 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:57:59.057467    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:58:09.431401 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.431302    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"]
Nov 02 18:58:09.432778 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.432749    4476 cpu_manager.go:414] "containerMap: removing container" podUID="2aca7966-d0ce-43cf-ae4a-8738531eb118" containerName="revoke-keys"
Nov 02 18:58:09.432950 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.432934    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aca7966-d0ce-43cf-ae4a-8738531eb118" containerName="revoke-keys"
Nov 02 18:58:09.433182 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.433165    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aca7966-d0ce-43cf-ae4a-8738531eb118" containerName="revoke-keys"
Nov 02 18:58:09.437812 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.437782    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.491556 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.491523    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"]
Nov 02 18:58:09.537525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.537487    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:58:09.537525 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.537488    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:58:09.537849 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.537695    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 18:58:09.557489 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.557454    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.557718 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.557546    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.557718 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.557591    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.591376 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.591345    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:58:09.658491 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.658440    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.658752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.658545    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.658752 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.658588    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.659879 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.659847    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.662310 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.662281    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.716140 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.716054    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t\") pod \"maas-api-revoke-sub-81ac00b3-kgmfl\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:09.761313 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:09.761271    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:10.192100 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:10.191137    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"]
Nov 02 18:58:10.207837 ip-10-0-136-192 kubenswrapper[4476]: W1102 18:58:10.207805    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa061ef_f0fe_4cfc_be76_5f0f77cac8ec.slice/crio-ee5a4d0743ec90be7cc5b47c197e58a825cef202d006087331f3b7adff9aa35a WatchSource:0}: Error finding container ee5a4d0743ec90be7cc5b47c197e58a825cef202d006087331f3b7adff9aa35a: Status 404 returned error can't find the container with id ee5a4d0743ec90be7cc5b47c197e58a825cef202d006087331f3b7adff9aa35a
Nov 02 18:58:10.522197 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:10.522127    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl" event={"ID":"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec","Type":"ContainerStarted","Data":"ee5a4d0743ec90be7cc5b47c197e58a825cef202d006087331f3b7adff9aa35a"}
Nov 02 18:58:11.529438 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:11.529401    4476 generic.go:356] "Generic (PLEG): container finished" podID="7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" containerID="058349b980f85cbf2e81bc3a39313c2532bd761d1ad1b153d5a334beb8b47c77" exitCode=0
Nov 02 18:58:11.529932 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:11.529526    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl" event={"ID":"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec","Type":"ContainerDied","Data":"058349b980f85cbf2e81bc3a39313c2532bd761d1ad1b153d5a334beb8b47c77"}
Nov 02 18:58:12.667701 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.667672    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:12.822316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.822241    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t\") pod \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") "
Nov 02 18:58:12.822316 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.822276    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca\") pod \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") "
Nov 02 18:58:12.822488 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.822318    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca\") pod \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\" (UID: \"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec\") "
Nov 02 18:58:12.822742 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.822713    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca" pod "7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" (UID: "7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:58:12.824526 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.824498    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca" pod "7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" (UID: "7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:58:12.824662 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.824642    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t" pod "7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" (UID: "7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec"). InnerVolumeSpecName "kube-api-access-sfl8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:58:12.923732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.923687    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-kube-api-access-sfl8t\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:12.923732 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.923731    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:12.923891 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:12.923745    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 18:58:13.541510 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:13.541467    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl" event={"ID":"7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec","Type":"ContainerDied","Data":"ee5a4d0743ec90be7cc5b47c197e58a825cef202d006087331f3b7adff9aa35a"}
Nov 02 18:58:13.541510 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:13.541496    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"
Nov 02 18:58:13.541510 ip-10-0-136-192 kubenswrapper[4476]: I1102 18:58:13.541517    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee5a4d0743ec90be7cc5b47c197e58a825cef202d006087331f3b7adff9aa35a"
Nov 02 18:58:29.061776 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:58:29.061735    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:58:59.058417 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:58:59.058376    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:59:29.058065 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:59:29.058029    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 18:59:59.065407 ip-10-0-136-192 kubenswrapper[4476]: E1102 18:59:59.065365    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:00:29.058410 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:00:29.058366    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:00:59.061622 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:00:59.061578    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:01:29.129694 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:01:29.129567    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:01:59.057882 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:01:59.057844    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:02:29.058463 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:02:29.058421    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:02:50.390720 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:02:50.390692    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:02:50.406137 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:02:50.406109    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:02:55.044648 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:02:55.044576    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"]
Nov 02 19:02:55.047451 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:02:55.047426    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b4b2f6-c987k"]
Nov 02 19:02:55.391667 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:02:55.391610    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aca7966-d0ce-43cf-ae4a-8738531eb118" path="/var/lib/kubelet/pods/2aca7966-d0ce-43cf-ae4a-8738531eb118/volumes"
Nov 02 19:02:59.058070 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:02:59.058031    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:03:13.048033 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:03:13.048002    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"]
Nov 02 19:03:13.096740 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:03:13.096709    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ac00b3-kgmfl"]
Nov 02 19:03:13.385313 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:03:13.385270    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" path="/var/lib/kubelet/pods/7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec/volumes"
Nov 02 19:03:29.058857 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:03:29.058818    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:03:52.056203 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:03:52.056171    4476 scope.go:122] "RemoveContainer" containerID="ad48ed76d8c6fa01fbd2db0389807ac938cc906dd72404ab238c8b1107b8d1d9"
Nov 02 19:03:59.058185 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:03:59.058146    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:04:29.058868 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:04:29.058787    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:04:52.084177 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:04:52.084134    4476 scope.go:122] "RemoveContainer" containerID="058349b980f85cbf2e81bc3a39313c2532bd761d1ad1b153d5a334beb8b47c77"
Nov 02 19:04:59.058062 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:04:59.058017    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:05:29.058552 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:05:29.058506    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:05:59.061511 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:05:59.061468    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:06:29.058304 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:06:29.058268    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:06:59.057850 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:06:59.057811    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:07:29.124398 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:07:29.124293    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:07:50.427805 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:07:50.427778    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:07:50.446172 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:07:50.446140    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:07:59.059237 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:07:59.059191    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:08:29.058798 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:08:29.058759    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:08:59.058614 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:08:59.058576    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:09:29.059078 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:09:29.059045    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:09:59.062533 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:09:59.062479    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:10:29.057878 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:10:29.057793    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:10:59.059499 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:10:59.059439    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:11:29.060478 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:11:29.060432    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:11:59.058001 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:11:59.057954    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:12:03.372367 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.372326    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"]
Nov 02 19:12:03.372940 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.372768    4476 cpu_manager.go:414] "containerMap: removing container" podUID="7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" containerName="revoke-keys"
Nov 02 19:12:03.372940 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.372781    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" containerName="revoke-keys"
Nov 02 19:12:03.372940 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.372850    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa061ef-f0fe-4cfc-be76-5f0f77cac8ec" containerName="revoke-keys"
Nov 02 19:12:03.376320 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.376296    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.431671 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.431614    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"]
Nov 02 19:12:03.477608 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.477571    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:12:03.477608 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.477572    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:12:03.477608 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.477581    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:12:03.477608 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.477585    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:12:03.505151 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.505116    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4ln\" (UniqueName: \"kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.505151 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.505153    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.505390 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.505273    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.606439 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.606404    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4ln\" (UniqueName: \"kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.606439 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.606441    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.606679 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.606497    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.607132 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.607112    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.609002 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.608984    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.663083 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.663011    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4ln\" (UniqueName: \"kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln\") pod \"maas-api-revoke-sub-3beb8614-9295l\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:03.689871 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:03.689837    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:04.062810 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:04.062784    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"]
Nov 02 19:12:04.064690 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:12:04.064661    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03f0c2c_fb90_4470_9892_2cd18f760bf7.slice/crio-ae0905cdf540a6888492ea6c33a83f3c57331b226527ac22d99db953bbecd2dd WatchSource:0}: Error finding container ae0905cdf540a6888492ea6c33a83f3c57331b226527ac22d99db953bbecd2dd: Status 404 returned error can't find the container with id ae0905cdf540a6888492ea6c33a83f3c57331b226527ac22d99db953bbecd2dd
Nov 02 19:12:04.730598 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:04.730562    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l" event={"ID":"d03f0c2c-fb90-4470-9892-2cd18f760bf7","Type":"ContainerStarted","Data":"91f38f29d39b639b24fb6bd88c06f2c474651ed0a7a3e5ad4cfb2c800205f576"}
Nov 02 19:12:04.731030 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:04.730619    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l" event={"ID":"d03f0c2c-fb90-4470-9892-2cd18f760bf7","Type":"ContainerStarted","Data":"ae0905cdf540a6888492ea6c33a83f3c57331b226527ac22d99db953bbecd2dd"}
Nov 02 19:12:04.842535 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:04.842477    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l" podStartSLOduration=1.842463518 podStartE2EDuration="1.842463518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:12:04.841794437 +0000 UTC m=+6555.682400371" watchObservedRunningTime="2026-11-02 19:12:04.842463518 +0000 UTC m=+6555.683069452"
Nov 02 19:12:05.737013 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:05.736978    4476 generic.go:356] "Generic (PLEG): container finished" podID="d03f0c2c-fb90-4470-9892-2cd18f760bf7" containerID="91f38f29d39b639b24fb6bd88c06f2c474651ed0a7a3e5ad4cfb2c800205f576" exitCode=0
Nov 02 19:12:05.737419 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:05.737063    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l" event={"ID":"d03f0c2c-fb90-4470-9892-2cd18f760bf7","Type":"ContainerDied","Data":"91f38f29d39b639b24fb6bd88c06f2c474651ed0a7a3e5ad4cfb2c800205f576"}
Nov 02 19:12:06.879052 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.879029    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:06.938256 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.938230    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca\") pod \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") "
Nov 02 19:12:06.938404 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.938281    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca\") pod \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") "
Nov 02 19:12:06.938404 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.938346    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln\" (UniqueName: \"kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln\") pod \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\" (UID: \"d03f0c2c-fb90-4470-9892-2cd18f760bf7\") "
Nov 02 19:12:06.938647 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.938598    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca" pod "d03f0c2c-fb90-4470-9892-2cd18f760bf7" (UID: "d03f0c2c-fb90-4470-9892-2cd18f760bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:12:06.940725 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.940698    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca" pod "d03f0c2c-fb90-4470-9892-2cd18f760bf7" (UID: "d03f0c2c-fb90-4470-9892-2cd18f760bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:12:06.940996 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:06.940973    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln" pod "d03f0c2c-fb90-4470-9892-2cd18f760bf7" (UID: "d03f0c2c-fb90-4470-9892-2cd18f760bf7"). InnerVolumeSpecName "kube-api-access-hm4ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:12:07.039263 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:07.039183    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4ln\" (UniqueName: \"kubernetes.io/projected/d03f0c2c-fb90-4470-9892-2cd18f760bf7-kube-api-access-hm4ln\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:07.039263 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:07.039215    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:07.039263 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:07.039231    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03f0c2c-fb90-4470-9892-2cd18f760bf7-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:12:07.750737 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:07.750657    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l" event={"ID":"d03f0c2c-fb90-4470-9892-2cd18f760bf7","Type":"ContainerDied","Data":"ae0905cdf540a6888492ea6c33a83f3c57331b226527ac22d99db953bbecd2dd"}
Nov 02 19:12:07.750737 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:07.750714    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae0905cdf540a6888492ea6c33a83f3c57331b226527ac22d99db953bbecd2dd"
Nov 02 19:12:07.750737 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:07.750666    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"
Nov 02 19:12:29.058768 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:12:29.058727    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:12:50.468488 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:50.468461    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:12:50.492971 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:12:50.492944    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:12:59.058238 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:12:59.058199    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:13:29.058948 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:13:29.058866    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:13:51.285569 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:51.285531    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"]
Nov 02 19:13:51.286127 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:51.285757    4476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g" podUID="f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" containerName="manager" containerID="cri-o://67b53cd7e142a73b679e518ce5cb86364311ac6676953f893e829945b7b6be25" gracePeriod=10
Nov 02 19:13:52.150075 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.149965    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 19:13:52.175970 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.175931    4476 scope.go:122] "RemoveContainer" containerID="67b53cd7e142a73b679e518ce5cb86364311ac6676953f893e829945b7b6be25"
Nov 02 19:13:52.199606 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.199585    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt\" (UniqueName: \"kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt\") pod \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") "
Nov 02 19:13:52.199700 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.199684    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume\") pod \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\" (UID: \"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6\") "
Nov 02 19:13:52.200075 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.200054    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume" pod "f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" (UID: "f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 19:13:52.201952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.201930    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt" pod "f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" (UID: "f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6"). InnerVolumeSpecName "kube-api-access-drrpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:13:52.301473 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.301389    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drrpt\" (UniqueName: \"kubernetes.io/projected/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-kube-api-access-drrpt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:52.301473 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.301420    4476 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6-extensions-socket-volume\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:13:52.340150 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.340110    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"
Nov 02 19:13:52.340362 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.340103    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g" event={"ID":"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6","Type":"ContainerDied","Data":"67b53cd7e142a73b679e518ce5cb86364311ac6676953f893e829945b7b6be25"}
Nov 02 19:13:52.340362 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.340250    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g" event={"ID":"f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6","Type":"ContainerDied","Data":"62db9ed842492fb3c0d5e356099564a63b584e210d50c7be3922145641a81c66"}
Nov 02 19:13:52.565665 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.565566    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"]
Nov 02 19:13:52.569408 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:52.569384    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtg8g"]
Nov 02 19:13:53.381302 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:13:53.381265    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" path="/var/lib/kubelet/pods/f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6/volumes"
Nov 02 19:13:59.058898 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:13:59.058864    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:14:29.058160 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:14:29.058116    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:14:54.861477 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861432    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"]
Nov 02 19:14:54.861952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861823    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d03f0c2c-fb90-4470-9892-2cd18f760bf7" containerName="revoke-keys"
Nov 02 19:14:54.861952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861836    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03f0c2c-fb90-4470-9892-2cd18f760bf7" containerName="revoke-keys"
Nov 02 19:14:54.861952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861863    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" containerName="manager"
Nov 02 19:14:54.861952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861869    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" containerName="manager"
Nov 02 19:14:54.861952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861927    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03f0c2c-fb90-4470-9892-2cd18f760bf7" containerName="revoke-keys"
Nov 02 19:14:54.861952 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.861939    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2275c4f-e7d1-4aa0-8612-1a7e3d2e7fb6" containerName="manager"
Nov 02 19:14:54.864987 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.864970    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:54.918412 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.918386    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"]
Nov 02 19:14:54.949651 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.949614    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65pd\" (UniqueName: \"kubernetes.io/projected/8b1fabde-e5ef-45af-9075-caef5e17ada1-kube-api-access-p65pd\") pod \"kuadrant-operator-controller-manager-678d87766f-2rbj7\" (UID: \"8b1fabde-e5ef-45af-9075-caef5e17ada1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:54.949821 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.949673    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1fabde-e5ef-45af-9075-caef5e17ada1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2rbj7\" (UID: \"8b1fabde-e5ef-45af-9075-caef5e17ada1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:54.965180 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:54.965154    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4rzcw\""
Nov 02 19:14:55.050928 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:55.050897    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65pd\" (UniqueName: \"kubernetes.io/projected/8b1fabde-e5ef-45af-9075-caef5e17ada1-kube-api-access-p65pd\") pod \"kuadrant-operator-controller-manager-678d87766f-2rbj7\" (UID: \"8b1fabde-e5ef-45af-9075-caef5e17ada1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:55.051097 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:55.050945    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1fabde-e5ef-45af-9075-caef5e17ada1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2rbj7\" (UID: \"8b1fabde-e5ef-45af-9075-caef5e17ada1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:55.051328 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:55.051311    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b1fabde-e5ef-45af-9075-caef5e17ada1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2rbj7\" (UID: \"8b1fabde-e5ef-45af-9075-caef5e17ada1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:55.107406 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:55.107379    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65pd\" (UniqueName: \"kubernetes.io/projected/8b1fabde-e5ef-45af-9075-caef5e17ada1-kube-api-access-p65pd\") pod \"kuadrant-operator-controller-manager-678d87766f-2rbj7\" (UID: \"8b1fabde-e5ef-45af-9075-caef5e17ada1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:55.177395 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:55.177320    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:55.660598 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:14:55.660570    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1fabde_e5ef_45af_9075_caef5e17ada1.slice/crio-5c71e0f689ac8f80163d8551d39ec3c4905fba877056836c05922425ff8fb19e WatchSource:0}: Error finding container 5c71e0f689ac8f80163d8551d39ec3c4905fba877056836c05922425ff8fb19e: Status 404 returned error can't find the container with id 5c71e0f689ac8f80163d8551d39ec3c4905fba877056836c05922425ff8fb19e
Nov 02 19:14:55.708433 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:55.708405    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"]
Nov 02 19:14:56.671350 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:56.671316    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7" event={"ID":"8b1fabde-e5ef-45af-9075-caef5e17ada1","Type":"ContainerStarted","Data":"94dca74737029d2a580a327aab7b3088c8daf4cbf0d1ccf7f9963f6d7d42aada"}
Nov 02 19:14:56.671814 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:56.671376    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7" event={"ID":"8b1fabde-e5ef-45af-9075-caef5e17ada1","Type":"ContainerStarted","Data":"5c71e0f689ac8f80163d8551d39ec3c4905fba877056836c05922425ff8fb19e"}
Nov 02 19:14:56.671814 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:56.671505    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:14:56.891190 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:14:56.891135    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7" podStartSLOduration=2.891120134 podStartE2EDuration="2.891120134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:14:56.839276984 +0000 UTC m=+6727.679882919" watchObservedRunningTime="2026-11-02 19:14:56.891120134 +0000 UTC m=+6727.731726068"
Nov 02 19:14:59.058749 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:14:59.058715    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:15:07.678850 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:15:07.678816    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2rbj7"
Nov 02 19:15:29.058820 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:15:29.058772    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:15:59.058962 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:15:59.058923    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:16:22.598404 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.598324    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"]
Nov 02 19:16:22.601704 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.601688    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.650846 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.650820    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.650994 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.650869    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.650994 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.650964    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwjs\" (UniqueName: \"kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.658061 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.658000    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"]
Nov 02 19:16:22.703835 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.703803    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:16:22.704009 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.703871    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:16:22.704009 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.703882    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:16:22.704089 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.704009    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:16:22.751951 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.751917    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwjs\" (UniqueName: \"kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.752103 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.752002    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.752103 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.752038    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.752726 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.752699    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.754530 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.754511    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.808164 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.808134    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwjs\" (UniqueName: \"kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs\") pod \"maas-api-revoke-sub-08a949cf-x2hf2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:22.914723 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:22.914686    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:23.287211 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:23.287184    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"]
Nov 02 19:16:23.290169 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:16:23.290141    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e87d3f_88ea_4a10_8e10_553cc86786a2.slice/crio-6321fea740a5f0fd67fc7e0369b387e60944275ca7e14bdbf995afc60c602378 WatchSource:0}: Error finding container 6321fea740a5f0fd67fc7e0369b387e60944275ca7e14bdbf995afc60c602378: Status 404 returned error can't find the container with id 6321fea740a5f0fd67fc7e0369b387e60944275ca7e14bdbf995afc60c602378
Nov 02 19:16:24.250043 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:24.250008    4476 generic.go:356] "Generic (PLEG): container finished" podID="f9e87d3f-88ea-4a10-8e10-553cc86786a2" containerID="dc7d4fc8edf9bb6acf21952e2f297b4736698fdab1f45d9fe2381db876d1e288" exitCode=0
Nov 02 19:16:24.250482 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:24.250095    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2" event={"ID":"f9e87d3f-88ea-4a10-8e10-553cc86786a2","Type":"ContainerDied","Data":"dc7d4fc8edf9bb6acf21952e2f297b4736698fdab1f45d9fe2381db876d1e288"}
Nov 02 19:16:24.250482 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:24.250145    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2" event={"ID":"f9e87d3f-88ea-4a10-8e10-553cc86786a2","Type":"ContainerStarted","Data":"6321fea740a5f0fd67fc7e0369b387e60944275ca7e14bdbf995afc60c602378"}
Nov 02 19:16:25.392866 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.392839    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:25.476253 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.476225    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca\") pod \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") "
Nov 02 19:16:25.476443 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.476271    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca\") pod \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") "
Nov 02 19:16:25.476443 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.476304    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs\" (UniqueName: \"kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs\") pod \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\" (UID: \"f9e87d3f-88ea-4a10-8e10-553cc86786a2\") "
Nov 02 19:16:25.476595 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.476570    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca" pod "f9e87d3f-88ea-4a10-8e10-553cc86786a2" (UID: "f9e87d3f-88ea-4a10-8e10-553cc86786a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:16:25.476738 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.476719    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:25.478509 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.478487    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca" pod "f9e87d3f-88ea-4a10-8e10-553cc86786a2" (UID: "f9e87d3f-88ea-4a10-8e10-553cc86786a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:16:25.478563 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.478519    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs" pod "f9e87d3f-88ea-4a10-8e10-553cc86786a2" (UID: "f9e87d3f-88ea-4a10-8e10-553cc86786a2"). InnerVolumeSpecName "kube-api-access-qmwjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:16:25.577720 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.577622    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e87d3f-88ea-4a10-8e10-553cc86786a2-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:25.577720 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:25.577668    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwjs\" (UniqueName: \"kubernetes.io/projected/f9e87d3f-88ea-4a10-8e10-553cc86786a2-kube-api-access-qmwjs\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:26.262506 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:26.262476    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"
Nov 02 19:16:26.262755 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:26.262469    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2" event={"ID":"f9e87d3f-88ea-4a10-8e10-553cc86786a2","Type":"ContainerDied","Data":"6321fea740a5f0fd67fc7e0369b387e60944275ca7e14bdbf995afc60c602378"}
Nov 02 19:16:26.262755 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:26.262598    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6321fea740a5f0fd67fc7e0369b387e60944275ca7e14bdbf995afc60c602378"
Nov 02 19:16:29.058674 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:16:29.058622    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:16:33.329681 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.329647    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"]
Nov 02 19:16:33.330345 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.330228    4476 cpu_manager.go:414] "containerMap: removing container" podUID="f9e87d3f-88ea-4a10-8e10-553cc86786a2" containerName="revoke-keys"
Nov 02 19:16:33.330345 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.330262    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e87d3f-88ea-4a10-8e10-553cc86786a2" containerName="revoke-keys"
Nov 02 19:16:33.330554 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.330371    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e87d3f-88ea-4a10-8e10-553cc86786a2" containerName="revoke-keys"
Nov 02 19:16:33.333722 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.333703    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.385932 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.385905    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:16:33.386099 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.385978    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:16:33.386150 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.386122    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:16:33.386216 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.386198    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:16:33.437322 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.437291    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"]
Nov 02 19:16:33.446344 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.446308    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5bz\" (UniqueName: \"kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.446524 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.446441    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.446524 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.446486    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.547644 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.547598    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5bz\" (UniqueName: \"kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.547824 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.547664    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.547824 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.547691    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.548333 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.548300    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.550322 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.550299    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.603471 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.603407    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5bz\" (UniqueName: \"kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz\") pod \"maas-api-revoke-sub-d65a4e6e-z7gt5\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:33.647514 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:33.647484    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:34.018429 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:34.018405    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"]
Nov 02 19:16:34.021287 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:16:34.021256    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b81b6c_104c_42a1_b977_b62c696d6b38.slice/crio-55c8ab4a06476e53409757bf1d0b63648f2b73359c888f8d582a8717c2debbf8 WatchSource:0}: Error finding container 55c8ab4a06476e53409757bf1d0b63648f2b73359c888f8d582a8717c2debbf8: Status 404 returned error can't find the container with id 55c8ab4a06476e53409757bf1d0b63648f2b73359c888f8d582a8717c2debbf8
Nov 02 19:16:34.313253 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:34.313169    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5" event={"ID":"80b81b6c-104c-42a1-b977-b62c696d6b38","Type":"ContainerStarted","Data":"55c8ab4a06476e53409757bf1d0b63648f2b73359c888f8d582a8717c2debbf8"}
Nov 02 19:16:35.320912 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:35.320880    4476 generic.go:356] "Generic (PLEG): container finished" podID="80b81b6c-104c-42a1-b977-b62c696d6b38" containerID="4d3c3b32f4be8ac872cba4fa74344bf0993cd4089d0e36d0ea96fdfc78069d46" exitCode=0
Nov 02 19:16:35.321322 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:35.320940    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5" event={"ID":"80b81b6c-104c-42a1-b977-b62c696d6b38","Type":"ContainerDied","Data":"4d3c3b32f4be8ac872cba4fa74344bf0993cd4089d0e36d0ea96fdfc78069d46"}
Nov 02 19:16:36.459802 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.459773    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:36.574681 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.574617    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca\") pod \"80b81b6c-104c-42a1-b977-b62c696d6b38\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") "
Nov 02 19:16:36.574930 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.574749    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca\") pod \"80b81b6c-104c-42a1-b977-b62c696d6b38\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") "
Nov 02 19:16:36.574930 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.574837    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz\" (UniqueName: \"kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz\") pod \"80b81b6c-104c-42a1-b977-b62c696d6b38\" (UID: \"80b81b6c-104c-42a1-b977-b62c696d6b38\") "
Nov 02 19:16:36.575115 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.575092    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca" pod "80b81b6c-104c-42a1-b977-b62c696d6b38" (UID: "80b81b6c-104c-42a1-b977-b62c696d6b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:16:36.577078 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.577024    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca" pod "80b81b6c-104c-42a1-b977-b62c696d6b38" (UID: "80b81b6c-104c-42a1-b977-b62c696d6b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:16:36.577371 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.577344    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz" pod "80b81b6c-104c-42a1-b977-b62c696d6b38" (UID: "80b81b6c-104c-42a1-b977-b62c696d6b38"). InnerVolumeSpecName "kube-api-access-7f5bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:16:36.676138 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.676101    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5bz\" (UniqueName: \"kubernetes.io/projected/80b81b6c-104c-42a1-b977-b62c696d6b38-kube-api-access-7f5bz\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:36.676138 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.676133    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:36.676138 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:36.676143    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b81b6c-104c-42a1-b977-b62c696d6b38-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:16:37.332394 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:37.332364    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"
Nov 02 19:16:37.332585 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:37.332402    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5" event={"ID":"80b81b6c-104c-42a1-b977-b62c696d6b38","Type":"ContainerDied","Data":"55c8ab4a06476e53409757bf1d0b63648f2b73359c888f8d582a8717c2debbf8"}
Nov 02 19:16:37.332585 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:16:37.332445    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c8ab4a06476e53409757bf1d0b63648f2b73359c888f8d582a8717c2debbf8"
Nov 02 19:16:59.058821 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:16:59.058778    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:17:06.754722 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.754675    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"]
Nov 02 19:17:06.755128 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.755109    4476 cpu_manager.go:414] "containerMap: removing container" podUID="80b81b6c-104c-42a1-b977-b62c696d6b38" containerName="revoke-keys"
Nov 02 19:17:06.755128 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.755123    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b81b6c-104c-42a1-b977-b62c696d6b38" containerName="revoke-keys"
Nov 02 19:17:06.755220 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.755209    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b81b6c-104c-42a1-b977-b62c696d6b38" containerName="revoke-keys"
Nov 02 19:17:06.758406 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.758386    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.815824 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.815782    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"]
Nov 02 19:17:06.861727 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.861699    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:17:06.861891 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.861835    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:17:06.861891 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.861848    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:17:06.861891 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.861879    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:17:06.881589 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.881563    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.881708 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.881657    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzkx\" (UniqueName: \"kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.881775 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.881761    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.982359 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.982329    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzkx\" (UniqueName: \"kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.982557 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.982406    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.982557 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.982430    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.983737 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.983704    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:06.986130 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:06.986099    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:07.039314 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.039251    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzkx\" (UniqueName: \"kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx\") pod \"maas-api-revoke-sub-51954f6a-7bvmw\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:07.051677 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.051652    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"]
Nov 02 19:17:07.053908 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.053888    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3beb8614-9295l"]
Nov 02 19:17:07.070921 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.070902    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:07.382057 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.382024    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03f0c2c-fb90-4470-9892-2cd18f760bf7" path="/var/lib/kubelet/pods/d03f0c2c-fb90-4470-9892-2cd18f760bf7/volumes"
Nov 02 19:17:07.446847 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.446825    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"]
Nov 02 19:17:07.448372 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:17:07.448348    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90eac55_6e54_4ff2_a1a3_58a5947e767f.slice/crio-71c930c35031ac62d48e95313de4fd31952c7c11ba6008ea9eaf1354b8fa07d8 WatchSource:0}: Error finding container 71c930c35031ac62d48e95313de4fd31952c7c11ba6008ea9eaf1354b8fa07d8: Status 404 returned error can't find the container with id 71c930c35031ac62d48e95313de4fd31952c7c11ba6008ea9eaf1354b8fa07d8
Nov 02 19:17:07.504786 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:07.504753    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw" event={"ID":"e90eac55-6e54-4ff2-a1a3-58a5947e767f","Type":"ContainerStarted","Data":"71c930c35031ac62d48e95313de4fd31952c7c11ba6008ea9eaf1354b8fa07d8"}
Nov 02 19:17:08.511806 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:08.511769    4476 generic.go:356] "Generic (PLEG): container finished" podID="e90eac55-6e54-4ff2-a1a3-58a5947e767f" containerID="a8c0088d7395a18f65d8ac96a47e0bdad47293a7b8f193d63e05fb5d05b2b3c8" exitCode=0
Nov 02 19:17:08.512274 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:08.511847    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw" event={"ID":"e90eac55-6e54-4ff2-a1a3-58a5947e767f","Type":"ContainerDied","Data":"a8c0088d7395a18f65d8ac96a47e0bdad47293a7b8f193d63e05fb5d05b2b3c8"}
Nov 02 19:17:09.649088 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.649064    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:09.707728 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.707693    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx\" (UniqueName: \"kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx\") pod \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") "
Nov 02 19:17:09.707909 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.707762    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca\") pod \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") "
Nov 02 19:17:09.707909 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.707818    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca\") pod \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\" (UID: \"e90eac55-6e54-4ff2-a1a3-58a5947e767f\") "
Nov 02 19:17:09.708131 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.708107    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca" pod "e90eac55-6e54-4ff2-a1a3-58a5947e767f" (UID: "e90eac55-6e54-4ff2-a1a3-58a5947e767f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:17:09.710075 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.710055    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca" pod "e90eac55-6e54-4ff2-a1a3-58a5947e767f" (UID: "e90eac55-6e54-4ff2-a1a3-58a5947e767f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:17:09.710168 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.710150    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx" pod "e90eac55-6e54-4ff2-a1a3-58a5947e767f" (UID: "e90eac55-6e54-4ff2-a1a3-58a5947e767f"). InnerVolumeSpecName "kube-api-access-cbzkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:17:09.808528 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.808456    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbzkx\" (UniqueName: \"kubernetes.io/projected/e90eac55-6e54-4ff2-a1a3-58a5947e767f-kube-api-access-cbzkx\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:17:09.808528 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.808483    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:17:09.808528 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:09.808492    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90eac55-6e54-4ff2-a1a3-58a5947e767f-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:17:10.523912 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:10.523878    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"
Nov 02 19:17:10.523912 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:10.523888    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw" event={"ID":"e90eac55-6e54-4ff2-a1a3-58a5947e767f","Type":"ContainerDied","Data":"71c930c35031ac62d48e95313de4fd31952c7c11ba6008ea9eaf1354b8fa07d8"}
Nov 02 19:17:10.524148 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:10.523937    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c930c35031ac62d48e95313de4fd31952c7c11ba6008ea9eaf1354b8fa07d8"
Nov 02 19:17:29.058616 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:17:29.058577    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:17:50.513673 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:50.513641    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:17:50.540734 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:50.540710    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:17:51.639412 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.639369    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"]
Nov 02 19:17:51.639847 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.639786    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e90eac55-6e54-4ff2-a1a3-58a5947e767f" containerName="revoke-keys"
Nov 02 19:17:51.639847 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.639797    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90eac55-6e54-4ff2-a1a3-58a5947e767f" containerName="revoke-keys"
Nov 02 19:17:51.639942 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.639868    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90eac55-6e54-4ff2-a1a3-58a5947e767f" containerName="revoke-keys"
Nov 02 19:17:51.643014 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.642995    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.698556 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.698524    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"]
Nov 02 19:17:51.739280 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.739253    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:17:51.744049 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.744027    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:17:51.744161 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.744122    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:17:51.744226 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.744183    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:17:51.792573 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.792542    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.792773 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.792612    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.792773 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.792659    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zrg\" (UniqueName: \"kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.893493 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.893406    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.893493 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.893471    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.893796 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.893500    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zrg\" (UniqueName: \"kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.894166 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.894145    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.896036 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.896015    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.949430 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.949395    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zrg\" (UniqueName: \"kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg\") pod \"maas-api-revoke-sub-659ba7af-h7fz4\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:51.956232 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:51.956209    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:52.333318 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:52.333293    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"]
Nov 02 19:17:52.335409 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:17:52.335377    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1b0b76_e4a1_42aa_ae91_12fdb1651188.slice/crio-0edffc89f86fb613a9b15fd36d0ed2b0201917cfd6556db5c27e97ae5d9608e4 WatchSource:0}: Error finding container 0edffc89f86fb613a9b15fd36d0ed2b0201917cfd6556db5c27e97ae5d9608e4: Status 404 returned error can't find the container with id 0edffc89f86fb613a9b15fd36d0ed2b0201917cfd6556db5c27e97ae5d9608e4
Nov 02 19:17:52.803166 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:52.803128    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4" event={"ID":"6f1b0b76-e4a1-42aa-ae91-12fdb1651188","Type":"ContainerStarted","Data":"04dbec4ef1c8bca3fcaededbaab4521f82b791632ecc09b7965437cd9e5b8ab8"}
Nov 02 19:17:52.803577 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:52.803192    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4" event={"ID":"6f1b0b76-e4a1-42aa-ae91-12fdb1651188","Type":"ContainerStarted","Data":"0edffc89f86fb613a9b15fd36d0ed2b0201917cfd6556db5c27e97ae5d9608e4"}
Nov 02 19:17:52.965726 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:52.965562    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4" podStartSLOduration=1.965547481 podStartE2EDuration="1.965547481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:17:52.914513673 +0000 UTC m=+6903.755119607" watchObservedRunningTime="2026-11-02 19:17:52.965547481 +0000 UTC m=+6903.806153414"
Nov 02 19:17:53.810612 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:53.810576    4476 generic.go:356] "Generic (PLEG): container finished" podID="6f1b0b76-e4a1-42aa-ae91-12fdb1651188" containerID="04dbec4ef1c8bca3fcaededbaab4521f82b791632ecc09b7965437cd9e5b8ab8" exitCode=0
Nov 02 19:17:53.811036 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:53.810622    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4" event={"ID":"6f1b0b76-e4a1-42aa-ae91-12fdb1651188","Type":"ContainerDied","Data":"04dbec4ef1c8bca3fcaededbaab4521f82b791632ecc09b7965437cd9e5b8ab8"}
Nov 02 19:17:54.949226 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:54.949202    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:55.020184 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.020158    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg\" (UniqueName: \"kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg\") pod \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") "
Nov 02 19:17:55.020324 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.020241    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca\") pod \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") "
Nov 02 19:17:55.020376 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.020335    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca\") pod \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\" (UID: \"6f1b0b76-e4a1-42aa-ae91-12fdb1651188\") "
Nov 02 19:17:55.020616 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.020588    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca" pod "6f1b0b76-e4a1-42aa-ae91-12fdb1651188" (UID: "6f1b0b76-e4a1-42aa-ae91-12fdb1651188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:17:55.022458 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.022430    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg" pod "6f1b0b76-e4a1-42aa-ae91-12fdb1651188" (UID: "6f1b0b76-e4a1-42aa-ae91-12fdb1651188"). InnerVolumeSpecName "kube-api-access-r5zrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:17:55.022565 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.022532    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca" pod "6f1b0b76-e4a1-42aa-ae91-12fdb1651188" (UID: "6f1b0b76-e4a1-42aa-ae91-12fdb1651188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:17:55.121127 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.121095    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:17:55.121127 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.121125    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:17:55.121335 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.121135    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zrg\" (UniqueName: \"kubernetes.io/projected/6f1b0b76-e4a1-42aa-ae91-12fdb1651188-kube-api-access-r5zrg\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:17:55.821833 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.821752    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4" event={"ID":"6f1b0b76-e4a1-42aa-ae91-12fdb1651188","Type":"ContainerDied","Data":"0edffc89f86fb613a9b15fd36d0ed2b0201917cfd6556db5c27e97ae5d9608e4"}
Nov 02 19:17:55.821833 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.821785    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"
Nov 02 19:17:55.821833 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:17:55.821802    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0edffc89f86fb613a9b15fd36d0ed2b0201917cfd6556db5c27e97ae5d9608e4"
Nov 02 19:17:59.058587 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:17:59.058551    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:18:02.430771 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.430736    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"]
Nov 02 19:18:02.431186 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.431171    4476 cpu_manager.go:414] "containerMap: removing container" podUID="6f1b0b76-e4a1-42aa-ae91-12fdb1651188" containerName="revoke-keys"
Nov 02 19:18:02.431263 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.431190    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1b0b76-e4a1-42aa-ae91-12fdb1651188" containerName="revoke-keys"
Nov 02 19:18:02.431328 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.431289    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1b0b76-e4a1-42aa-ae91-12fdb1651188" containerName="revoke-keys"
Nov 02 19:18:02.434505 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.434485    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.490230 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.490192    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"]
Nov 02 19:18:02.535955 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.535911    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:18:02.535955 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.535921    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:18:02.536592 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.536569    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:18:02.536592 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.536585    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:18:02.591214 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.591175    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7glt\" (UniqueName: \"kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.591436 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.591244    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.591436 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.591368    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.692824 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.692735    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.692824 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.692794    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7glt\" (UniqueName: \"kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.693037 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.692849    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.693535 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.693465    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.695323 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.695298    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:02.749529 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:02.749499    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7glt\" (UniqueName: \"kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt\") pod \"maas-api-revoke-sub-ee0cb899-mclc2\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:03.050007 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:03.049918    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:03.424879 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:03.424852    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"]
Nov 02 19:18:03.426726 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:18:03.426701    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5e36b0_5984_43c4_bbec_c045bea98184.slice/crio-c97d864689f6b1c4014b82b52a02571085538fd110fd93aee30b43703057d3af WatchSource:0}: Error finding container c97d864689f6b1c4014b82b52a02571085538fd110fd93aee30b43703057d3af: Status 404 returned error can't find the container with id c97d864689f6b1c4014b82b52a02571085538fd110fd93aee30b43703057d3af
Nov 02 19:18:03.868674 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:03.868606    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2" event={"ID":"4e5e36b0-5984-43c4-bbec-c045bea98184","Type":"ContainerStarted","Data":"d2d1ee86eff7609c08df33e44146d9696606049b250ea32ba2efc2909813d7cb"}
Nov 02 19:18:03.869082 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:03.868707    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2" event={"ID":"4e5e36b0-5984-43c4-bbec-c045bea98184","Type":"ContainerStarted","Data":"c97d864689f6b1c4014b82b52a02571085538fd110fd93aee30b43703057d3af"}
Nov 02 19:18:03.982929 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:03.982860    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2" podStartSLOduration=1.982839458 podStartE2EDuration="1.982839458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:18:03.979216251 +0000 UTC m=+6914.819822185" watchObservedRunningTime="2026-11-02 19:18:03.982839458 +0000 UTC m=+6914.823445394"
Nov 02 19:18:04.876056 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:04.876017    4476 generic.go:356] "Generic (PLEG): container finished" podID="4e5e36b0-5984-43c4-bbec-c045bea98184" containerID="d2d1ee86eff7609c08df33e44146d9696606049b250ea32ba2efc2909813d7cb" exitCode=0
Nov 02 19:18:04.876435 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:04.876110    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2" event={"ID":"4e5e36b0-5984-43c4-bbec-c045bea98184","Type":"ContainerDied","Data":"d2d1ee86eff7609c08df33e44146d9696606049b250ea32ba2efc2909813d7cb"}
Nov 02 19:18:06.019471 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.019445    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:06.126815 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.126783    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca\") pod \"4e5e36b0-5984-43c4-bbec-c045bea98184\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") "
Nov 02 19:18:06.126815 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.126819    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt\" (UniqueName: \"kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt\") pod \"4e5e36b0-5984-43c4-bbec-c045bea98184\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") "
Nov 02 19:18:06.127031 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.126859    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca\") pod \"4e5e36b0-5984-43c4-bbec-c045bea98184\" (UID: \"4e5e36b0-5984-43c4-bbec-c045bea98184\") "
Nov 02 19:18:06.127277 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.127253    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca" pod "4e5e36b0-5984-43c4-bbec-c045bea98184" (UID: "4e5e36b0-5984-43c4-bbec-c045bea98184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:18:06.129184 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.129157    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca" pod "4e5e36b0-5984-43c4-bbec-c045bea98184" (UID: "4e5e36b0-5984-43c4-bbec-c045bea98184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:18:06.129296 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.129238    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt" pod "4e5e36b0-5984-43c4-bbec-c045bea98184" (UID: "4e5e36b0-5984-43c4-bbec-c045bea98184"). InnerVolumeSpecName "kube-api-access-x7glt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:18:06.227612 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.227580    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:18:06.227612 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.227607    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7glt\" (UniqueName: \"kubernetes.io/projected/4e5e36b0-5984-43c4-bbec-c045bea98184-kube-api-access-x7glt\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:18:06.227612 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.227619    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e36b0-5984-43c4-bbec-c045bea98184-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:18:06.889175 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.889138    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"
Nov 02 19:18:06.889387 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.889137    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2" event={"ID":"4e5e36b0-5984-43c4-bbec-c045bea98184","Type":"ContainerDied","Data":"c97d864689f6b1c4014b82b52a02571085538fd110fd93aee30b43703057d3af"}
Nov 02 19:18:06.889387 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:06.889264    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c97d864689f6b1c4014b82b52a02571085538fd110fd93aee30b43703057d3af"
Nov 02 19:18:29.062082 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:18:29.062039    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:18:47.444110 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.444076    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"]
Nov 02 19:18:47.444535 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.444457    4476 cpu_manager.go:414] "containerMap: removing container" podUID="4e5e36b0-5984-43c4-bbec-c045bea98184" containerName="revoke-keys"
Nov 02 19:18:47.444535 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.444468    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5e36b0-5984-43c4-bbec-c045bea98184" containerName="revoke-keys"
Nov 02 19:18:47.444535 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.444524    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5e36b0-5984-43c4-bbec-c045bea98184" containerName="revoke-keys"
Nov 02 19:18:47.447428 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.447413    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.482387 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.482362    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr47s\" (UniqueName: \"kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.482526 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.482409    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.482526 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.482476    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.499793 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.499764    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:18:47.503168 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.503143    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"]
Nov 02 19:18:47.549023 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.548990    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:18:47.549023 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.549012    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:18:47.549023 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.548990    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:18:47.583538 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.583509    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr47s\" (UniqueName: \"kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.583697 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.583559    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.583697 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.583586    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.584312 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.584286    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.586157 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.586139    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.639992 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.639967    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr47s\" (UniqueName: \"kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s\") pod \"maas-api-revoke-sub-20ab545f-r5gvw\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:47.760086 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:47.760011    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:48.135757 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:48.135720    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"]
Nov 02 19:18:48.138008 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:18:48.137981    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52694be7_697d_44f3_9270_c54675228da4.slice/crio-225e9b47c2135173eb848285cd1719b7a8a5672bd1f8855958b32363ea0370d4 WatchSource:0}: Error finding container 225e9b47c2135173eb848285cd1719b7a8a5672bd1f8855958b32363ea0370d4: Status 404 returned error can't find the container with id 225e9b47c2135173eb848285cd1719b7a8a5672bd1f8855958b32363ea0370d4
Nov 02 19:18:48.183374 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:48.183347    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw" event={"ID":"52694be7-697d-44f3-9270-c54675228da4","Type":"ContainerStarted","Data":"225e9b47c2135173eb848285cd1719b7a8a5672bd1f8855958b32363ea0370d4"}
Nov 02 19:18:49.191485 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:49.191402    4476 generic.go:356] "Generic (PLEG): container finished" podID="52694be7-697d-44f3-9270-c54675228da4" containerID="c28baa16d2ce6ac941a103350dcc8e4859addc7b000882492aa56427f46e88b9" exitCode=0
Nov 02 19:18:49.191485 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:49.191450    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw" event={"ID":"52694be7-697d-44f3-9270-c54675228da4","Type":"ContainerDied","Data":"c28baa16d2ce6ac941a103350dcc8e4859addc7b000882492aa56427f46e88b9"}
Nov 02 19:18:50.338944 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.338921    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:50.408096 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.408065    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca\") pod \"52694be7-697d-44f3-9270-c54675228da4\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") "
Nov 02 19:18:50.408247 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.408116    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca\") pod \"52694be7-697d-44f3-9270-c54675228da4\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") "
Nov 02 19:18:50.408321 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.408251    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s\" (UniqueName: \"kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s\") pod \"52694be7-697d-44f3-9270-c54675228da4\" (UID: \"52694be7-697d-44f3-9270-c54675228da4\") "
Nov 02 19:18:50.408489 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.408467    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca" pod "52694be7-697d-44f3-9270-c54675228da4" (UID: "52694be7-697d-44f3-9270-c54675228da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:18:50.408613 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.408595    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52694be7-697d-44f3-9270-c54675228da4-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:18:50.410300 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.410274    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca" pod "52694be7-697d-44f3-9270-c54675228da4" (UID: "52694be7-697d-44f3-9270-c54675228da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:18:50.410405 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.410390    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s" pod "52694be7-697d-44f3-9270-c54675228da4" (UID: "52694be7-697d-44f3-9270-c54675228da4"). InnerVolumeSpecName "kube-api-access-rr47s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:18:50.509324 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.509243    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52694be7-697d-44f3-9270-c54675228da4-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:18:50.509324 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:50.509275    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr47s\" (UniqueName: \"kubernetes.io/projected/52694be7-697d-44f3-9270-c54675228da4-kube-api-access-rr47s\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:18:51.204713 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:51.204683    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw"
Nov 02 19:18:51.204713 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:51.204696    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ab545f-r5gvw" event={"ID":"52694be7-697d-44f3-9270-c54675228da4","Type":"ContainerDied","Data":"225e9b47c2135173eb848285cd1719b7a8a5672bd1f8855958b32363ea0370d4"}
Nov 02 19:18:51.204928 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:51.204746    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="225e9b47c2135173eb848285cd1719b7a8a5672bd1f8855958b32363ea0370d4"
Nov 02 19:18:52.238290 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:52.238259    4476 scope.go:122] "RemoveContainer" containerID="91f38f29d39b639b24fb6bd88c06f2c474651ed0a7a3e5ad4cfb2c800205f576"
Nov 02 19:18:58.245879 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.245832    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"]
Nov 02 19:18:58.246420 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.246398    4476 cpu_manager.go:414] "containerMap: removing container" podUID="52694be7-697d-44f3-9270-c54675228da4" containerName="revoke-keys"
Nov 02 19:18:58.246501 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.246424    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52694be7-697d-44f3-9270-c54675228da4" containerName="revoke-keys"
Nov 02 19:18:58.246578 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.246563    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="52694be7-697d-44f3-9270-c54675228da4" containerName="revoke-keys"
Nov 02 19:18:58.250984 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.250960    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.305556 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.305499    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"]
Nov 02 19:18:58.350219 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.350184    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:18:58.350219 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.350203    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:18:58.350476 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.350241    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:18:58.352487 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.352469    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:18:58.376089 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.376051    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.376089 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.376092    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.376282 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.376225    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdgh\" (UniqueName: \"kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.477170 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.477136    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgh\" (UniqueName: \"kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.477381 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.477194    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.477381 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.477251    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.477921 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.477898    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.479752 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.479731    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.533213 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.533144    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdgh\" (UniqueName: \"kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh\") pod \"maas-api-revoke-sub-d56aca28-nsnz2\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.564122 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.564087    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:18:58.937169 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:58.937142    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"]
Nov 02 19:18:58.939355 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:18:58.939325    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72781957_d260_4abe_a426_17d5461cf259.slice/crio-76c63a0ef997cbb9bc9088542c9283b5d30ee898fc9e9741aaca73c260102744 WatchSource:0}: Error finding container 76c63a0ef997cbb9bc9088542c9283b5d30ee898fc9e9741aaca73c260102744: Status 404 returned error can't find the container with id 76c63a0ef997cbb9bc9088542c9283b5d30ee898fc9e9741aaca73c260102744
Nov 02 19:18:59.059060 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:18:59.059016    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:18:59.258682 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:59.258620    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2" event={"ID":"72781957-d260-4abe-a426-17d5461cf259","Type":"ContainerStarted","Data":"7bd82093020ca430a711caeae83ed91a86dba57d9eed0f38705d6fef7231a2fe"}
Nov 02 19:18:59.259052 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:59.258704    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2" event={"ID":"72781957-d260-4abe-a426-17d5461cf259","Type":"ContainerStarted","Data":"76c63a0ef997cbb9bc9088542c9283b5d30ee898fc9e9741aaca73c260102744"}
Nov 02 19:18:59.370129 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:18:59.370069    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2" podStartSLOduration=1.370054182 podStartE2EDuration="1.370054182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:18:59.369233445 +0000 UTC m=+6970.209839391" watchObservedRunningTime="2026-11-02 19:18:59.370054182 +0000 UTC m=+6970.210660116"
Nov 02 19:19:00.266859 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:00.266820    4476 generic.go:356] "Generic (PLEG): container finished" podID="72781957-d260-4abe-a426-17d5461cf259" containerID="7bd82093020ca430a711caeae83ed91a86dba57d9eed0f38705d6fef7231a2fe" exitCode=0
Nov 02 19:19:00.267367 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:00.266898    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2" event={"ID":"72781957-d260-4abe-a426-17d5461cf259","Type":"ContainerDied","Data":"7bd82093020ca430a711caeae83ed91a86dba57d9eed0f38705d6fef7231a2fe"}
Nov 02 19:19:01.454726 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.454692    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:19:01.514663 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.513956    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca\") pod \"72781957-d260-4abe-a426-17d5461cf259\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") "
Nov 02 19:19:01.514663 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.514043    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh\" (UniqueName: \"kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh\") pod \"72781957-d260-4abe-a426-17d5461cf259\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") "
Nov 02 19:19:01.514663 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.514091    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca\") pod \"72781957-d260-4abe-a426-17d5461cf259\" (UID: \"72781957-d260-4abe-a426-17d5461cf259\") "
Nov 02 19:19:01.516951 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.516909    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca" pod "72781957-d260-4abe-a426-17d5461cf259" (UID: "72781957-d260-4abe-a426-17d5461cf259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:19:01.533481 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.533444    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca" pod "72781957-d260-4abe-a426-17d5461cf259" (UID: "72781957-d260-4abe-a426-17d5461cf259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:19:01.533481 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.533464    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh" pod "72781957-d260-4abe-a426-17d5461cf259" (UID: "72781957-d260-4abe-a426-17d5461cf259"). InnerVolumeSpecName "kube-api-access-wkdgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:19:01.616707 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.615242    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72781957-d260-4abe-a426-17d5461cf259-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:19:01.616707 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.615290    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdgh\" (UniqueName: \"kubernetes.io/projected/72781957-d260-4abe-a426-17d5461cf259-kube-api-access-wkdgh\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:19:01.616707 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:01.615311    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72781957-d260-4abe-a426-17d5461cf259-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:19:02.309753 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:02.309712    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2" event={"ID":"72781957-d260-4abe-a426-17d5461cf259","Type":"ContainerDied","Data":"76c63a0ef997cbb9bc9088542c9283b5d30ee898fc9e9741aaca73c260102744"}
Nov 02 19:19:02.310044 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:02.310025    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c63a0ef997cbb9bc9088542c9283b5d30ee898fc9e9741aaca73c260102744"
Nov 02 19:19:02.310237 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:02.310221    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d56aca28-nsnz2"
Nov 02 19:19:29.058743 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:19:29.058704    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:19:36.570876 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:36.570828    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:19:36.571738 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:36.571671    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:19:49.554445 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.554410    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"]
Nov 02 19:19:49.554884 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.554863    4476 cpu_manager.go:414] "containerMap: removing container" podUID="72781957-d260-4abe-a426-17d5461cf259" containerName="revoke-keys"
Nov 02 19:19:49.554884 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.554876    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72781957-d260-4abe-a426-17d5461cf259" containerName="revoke-keys"
Nov 02 19:19:49.554975 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.554944    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="72781957-d260-4abe-a426-17d5461cf259" containerName="revoke-keys"
Nov 02 19:19:49.558006 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.557990    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.613085 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.613051    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"]
Nov 02 19:19:49.659130 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659090    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:19:49.659307 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659090    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:19:49.659307 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659101    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:19:49.659307 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659177    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.659307 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659215    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.659457 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659329    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wx99\" (UniqueName: \"kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.659587 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.659569    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:19:49.760676 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.760618    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wx99\" (UniqueName: \"kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.760883 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.760716    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.760883 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.760745    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.761367 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.761346    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.763336 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.763310    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.817205 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.817126    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wx99\" (UniqueName: \"kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99\") pod \"maas-api-revoke-sub-2daade54-vvmqb\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:49.871048 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:49.871012    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:50.244196 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:50.244166    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"]
Nov 02 19:19:50.246123 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:19:50.246097    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae553987_3811_4165_8e37_8be1e15f82da.slice/crio-61ea374c5afe9de99cb585c3189f9de7c2b3ebd318b35da926f8c29b652038f4 WatchSource:0}: Error finding container 61ea374c5afe9de99cb585c3189f9de7c2b3ebd318b35da926f8c29b652038f4: Status 404 returned error can't find the container with id 61ea374c5afe9de99cb585c3189f9de7c2b3ebd318b35da926f8c29b652038f4
Nov 02 19:19:50.753790 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:50.753756    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb" event={"ID":"ae553987-3811-4165-8e37-8be1e15f82da","Type":"ContainerStarted","Data":"ad87a49c4936abfa7386b7f07a09800447a2d31eddc339941faff07714a15dbc"}
Nov 02 19:19:50.754242 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:50.753808    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb" event={"ID":"ae553987-3811-4165-8e37-8be1e15f82da","Type":"ContainerStarted","Data":"61ea374c5afe9de99cb585c3189f9de7c2b3ebd318b35da926f8c29b652038f4"}
Nov 02 19:19:50.867826 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:50.867776    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb" podStartSLOduration=1.86776257 podStartE2EDuration="1.86776257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:19:50.867536293 +0000 UTC m=+7021.708142239" watchObservedRunningTime="2026-11-02 19:19:50.86776257 +0000 UTC m=+7021.708368481"
Nov 02 19:19:51.762849 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:51.762812    4476 generic.go:356] "Generic (PLEG): container finished" podID="ae553987-3811-4165-8e37-8be1e15f82da" containerID="ad87a49c4936abfa7386b7f07a09800447a2d31eddc339941faff07714a15dbc" exitCode=0
Nov 02 19:19:51.763263 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:51.762897    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb" event={"ID":"ae553987-3811-4165-8e37-8be1e15f82da","Type":"ContainerDied","Data":"ad87a49c4936abfa7386b7f07a09800447a2d31eddc339941faff07714a15dbc"}
Nov 02 19:19:52.912528 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:52.912499    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:52.983224 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:52.983189    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99\" (UniqueName: \"kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99\") pod \"ae553987-3811-4165-8e37-8be1e15f82da\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") "
Nov 02 19:19:52.983411 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:52.983235    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca\") pod \"ae553987-3811-4165-8e37-8be1e15f82da\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") "
Nov 02 19:19:52.985565 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:52.985526    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca" pod "ae553987-3811-4165-8e37-8be1e15f82da" (UID: "ae553987-3811-4165-8e37-8be1e15f82da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:19:52.985717 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:52.985592    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99" pod "ae553987-3811-4165-8e37-8be1e15f82da" (UID: "ae553987-3811-4165-8e37-8be1e15f82da"). InnerVolumeSpecName "kube-api-access-5wx99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:19:53.084345 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.084262    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca\") pod \"ae553987-3811-4165-8e37-8be1e15f82da\" (UID: \"ae553987-3811-4165-8e37-8be1e15f82da\") "
Nov 02 19:19:53.084523 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.084422    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wx99\" (UniqueName: \"kubernetes.io/projected/ae553987-3811-4165-8e37-8be1e15f82da-kube-api-access-5wx99\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:19:53.084523 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.084435    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:19:53.084691 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.084670    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca" pod "ae553987-3811-4165-8e37-8be1e15f82da" (UID: "ae553987-3811-4165-8e37-8be1e15f82da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:19:53.185381 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.185343    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae553987-3811-4165-8e37-8be1e15f82da-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:19:53.775615 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.775530    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb"
Nov 02 19:19:53.775615 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.775559    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2daade54-vvmqb" event={"ID":"ae553987-3811-4165-8e37-8be1e15f82da","Type":"ContainerDied","Data":"61ea374c5afe9de99cb585c3189f9de7c2b3ebd318b35da926f8c29b652038f4"}
Nov 02 19:19:53.775615 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:19:53.775614    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ea374c5afe9de99cb585c3189f9de7c2b3ebd318b35da926f8c29b652038f4"
Nov 02 19:19:59.060811 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:19:59.060755    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:20:00.556347 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.556311    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"]
Nov 02 19:20:00.556767 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.556707    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ae553987-3811-4165-8e37-8be1e15f82da" containerName="revoke-keys"
Nov 02 19:20:00.556767 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.556719    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae553987-3811-4165-8e37-8be1e15f82da" containerName="revoke-keys"
Nov 02 19:20:00.556852 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.556804    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae553987-3811-4165-8e37-8be1e15f82da" containerName="revoke-keys"
Nov 02 19:20:00.560062 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.560042    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.662219 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662183    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:20:00.662219 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662183    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:20:00.662463 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662243    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:20:00.662463 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662269    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.662463 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662291    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:20:00.662463 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662394    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.662600 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.662505    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjkcw\" (UniqueName: \"kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.665202 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.665180    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"]
Nov 02 19:20:00.763534 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.763498    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.763754 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.763556    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjkcw\" (UniqueName: \"kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.763754 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.763609    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.764261 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.764242    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.766285 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.766262    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.820165 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.820090    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjkcw\" (UniqueName: \"kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw\") pod \"maas-api-revoke-sub-ac5a7d6c-rlzmt\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:00.873523 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:00.873486    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:01.256495 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:01.256464    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"]
Nov 02 19:20:01.258154 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:20:01.258127    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885728d4_e83d_4da3_b8e7_1bd1ab32bded.slice/crio-6e30fe01e0556abb36f3434dd282b0c7b17bf4cb458170daf0350a0588154a40 WatchSource:0}: Error finding container 6e30fe01e0556abb36f3434dd282b0c7b17bf4cb458170daf0350a0588154a40: Status 404 returned error can't find the container with id 6e30fe01e0556abb36f3434dd282b0c7b17bf4cb458170daf0350a0588154a40
Nov 02 19:20:01.823376 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:01.823295    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt" event={"ID":"885728d4-e83d-4da3-b8e7-1bd1ab32bded","Type":"ContainerStarted","Data":"8bcf94c8369898864f309e8fad9d542087a86863faf096350a651b3a1e543724"}
Nov 02 19:20:01.823376 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:01.823344    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt" event={"ID":"885728d4-e83d-4da3-b8e7-1bd1ab32bded","Type":"ContainerStarted","Data":"6e30fe01e0556abb36f3434dd282b0c7b17bf4cb458170daf0350a0588154a40"}
Nov 02 19:20:01.935496 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:01.935441    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt" podStartSLOduration=1.935426755 podStartE2EDuration="1.935426755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:20:01.933928077 +0000 UTC m=+7032.774534011" watchObservedRunningTime="2026-11-02 19:20:01.935426755 +0000 UTC m=+7032.776032709"
Nov 02 19:20:02.831292 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:02.831250    4476 generic.go:356] "Generic (PLEG): container finished" podID="885728d4-e83d-4da3-b8e7-1bd1ab32bded" containerID="8bcf94c8369898864f309e8fad9d542087a86863faf096350a651b3a1e543724" exitCode=0
Nov 02 19:20:02.831751 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:02.831334    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt" event={"ID":"885728d4-e83d-4da3-b8e7-1bd1ab32bded","Type":"ContainerDied","Data":"8bcf94c8369898864f309e8fad9d542087a86863faf096350a651b3a1e543724"}
Nov 02 19:20:03.980500 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:03.980472    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:04.100081 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.099999    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw\" (UniqueName: \"kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw\") pod \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") "
Nov 02 19:20:04.100248 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.100124    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca\") pod \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") "
Nov 02 19:20:04.100248 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.100161    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca\") pod \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\" (UID: \"885728d4-e83d-4da3-b8e7-1bd1ab32bded\") "
Nov 02 19:20:04.100486 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.100462    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca" pod "885728d4-e83d-4da3-b8e7-1bd1ab32bded" (UID: "885728d4-e83d-4da3-b8e7-1bd1ab32bded"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:20:04.102590 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.102560    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca" pod "885728d4-e83d-4da3-b8e7-1bd1ab32bded" (UID: "885728d4-e83d-4da3-b8e7-1bd1ab32bded"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:20:04.102796 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.102607    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw" pod "885728d4-e83d-4da3-b8e7-1bd1ab32bded" (UID: "885728d4-e83d-4da3-b8e7-1bd1ab32bded"). InnerVolumeSpecName "kube-api-access-rjkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:20:04.201177 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.201138    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:04.201177 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.201170    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/885728d4-e83d-4da3-b8e7-1bd1ab32bded-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:04.201177 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.201180    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjkcw\" (UniqueName: \"kubernetes.io/projected/885728d4-e83d-4da3-b8e7-1bd1ab32bded-kube-api-access-rjkcw\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:04.846677 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.846618    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt"
Nov 02 19:20:04.846866 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.846617    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5a7d6c-rlzmt" event={"ID":"885728d4-e83d-4da3-b8e7-1bd1ab32bded","Type":"ContainerDied","Data":"6e30fe01e0556abb36f3434dd282b0c7b17bf4cb458170daf0350a0588154a40"}
Nov 02 19:20:04.846866 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:04.846787    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e30fe01e0556abb36f3434dd282b0c7b17bf4cb458170daf0350a0588154a40"
Nov 02 19:20:06.568785 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:06.568746    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-thm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:20:06.569465 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:06.569429    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-thm4f" podUID="792eac8f-2598-4b49-94d1-6d0231efb074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:20:11.242646 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.242591    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"]
Nov 02 19:20:11.243279 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.243224    4476 cpu_manager.go:414] "containerMap: removing container" podUID="885728d4-e83d-4da3-b8e7-1bd1ab32bded" containerName="revoke-keys"
Nov 02 19:20:11.243279 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.243246    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885728d4-e83d-4da3-b8e7-1bd1ab32bded" containerName="revoke-keys"
Nov 02 19:20:11.243414 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.243375    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="885728d4-e83d-4da3-b8e7-1bd1ab32bded" containerName="revoke-keys"
Nov 02 19:20:11.246576 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.246548    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.262276 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.262247    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.262429 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.262334    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9qj\" (UniqueName: \"kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.262429 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.262418    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.303282 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.303245    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"]
Nov 02 19:20:11.347460 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.347426    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:20:11.347460 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.347451    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:20:11.347695 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.347504    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:20:11.347761 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.347747    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:20:11.363557 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.363524    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9qj\" (UniqueName: \"kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.363785 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.363571    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.363785 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.363651    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.364269 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.364249    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.366171 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.366146    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.422398 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.422368    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9qj\" (UniqueName: \"kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj\") pod \"maas-api-revoke-sub-fb0b5d82-2ffpl\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.562796 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.562717    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:11.938658 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:11.938607    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"]
Nov 02 19:20:11.939479 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:20:11.939456    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b84b0ca_c62f_402a_9123_c8664d3ea056.slice/crio-c031ea7cf9d136d990b78d3046cc7239d786afd24342576e13da89e51ba6360f WatchSource:0}: Error finding container c031ea7cf9d136d990b78d3046cc7239d786afd24342576e13da89e51ba6360f: Status 404 returned error can't find the container with id c031ea7cf9d136d990b78d3046cc7239d786afd24342576e13da89e51ba6360f
Nov 02 19:20:12.910785 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:12.910753    4476 generic.go:356] "Generic (PLEG): container finished" podID="0b84b0ca-c62f-402a-9123-c8664d3ea056" containerID="bb0789a963ea36f8d6af7e71c9321f829dfac2154669912cba6396f2d3a8e4a2" exitCode=0
Nov 02 19:20:12.911158 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:12.910837    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl" event={"ID":"0b84b0ca-c62f-402a-9123-c8664d3ea056","Type":"ContainerDied","Data":"bb0789a963ea36f8d6af7e71c9321f829dfac2154669912cba6396f2d3a8e4a2"}
Nov 02 19:20:12.911158 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:12.910892    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl" event={"ID":"0b84b0ca-c62f-402a-9123-c8664d3ea056","Type":"ContainerStarted","Data":"c031ea7cf9d136d990b78d3046cc7239d786afd24342576e13da89e51ba6360f"}
Nov 02 19:20:14.050611 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.050588    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:14.089910 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.089880    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca\") pod \"0b84b0ca-c62f-402a-9123-c8664d3ea056\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") "
Nov 02 19:20:14.090091 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.089921    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj\" (UniqueName: \"kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj\") pod \"0b84b0ca-c62f-402a-9123-c8664d3ea056\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") "
Nov 02 19:20:14.090091 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.089951    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca\") pod \"0b84b0ca-c62f-402a-9123-c8664d3ea056\" (UID: \"0b84b0ca-c62f-402a-9123-c8664d3ea056\") "
Nov 02 19:20:14.090305 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.090281    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca" pod "0b84b0ca-c62f-402a-9123-c8664d3ea056" (UID: "0b84b0ca-c62f-402a-9123-c8664d3ea056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:20:14.092152 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.092126    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj" pod "0b84b0ca-c62f-402a-9123-c8664d3ea056" (UID: "0b84b0ca-c62f-402a-9123-c8664d3ea056"). InnerVolumeSpecName "kube-api-access-xh9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:20:14.092274 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.092197    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca" pod "0b84b0ca-c62f-402a-9123-c8664d3ea056" (UID: "0b84b0ca-c62f-402a-9123-c8664d3ea056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:20:14.191281 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.191210    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:14.191281 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.191238    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9qj\" (UniqueName: \"kubernetes.io/projected/0b84b0ca-c62f-402a-9123-c8664d3ea056-kube-api-access-xh9qj\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:14.191281 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.191251    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b84b0ca-c62f-402a-9123-c8664d3ea056-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:14.923199 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.923171    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl"
Nov 02 19:20:14.923387 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.923169    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0b5d82-2ffpl" event={"ID":"0b84b0ca-c62f-402a-9123-c8664d3ea056","Type":"ContainerDied","Data":"c031ea7cf9d136d990b78d3046cc7239d786afd24342576e13da89e51ba6360f"}
Nov 02 19:20:14.923387 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:14.923295    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c031ea7cf9d136d990b78d3046cc7239d786afd24342576e13da89e51ba6360f"
Nov 02 19:20:22.406779 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.406743    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"]
Nov 02 19:20:22.407206 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.407131    4476 cpu_manager.go:414] "containerMap: removing container" podUID="0b84b0ca-c62f-402a-9123-c8664d3ea056" containerName="revoke-keys"
Nov 02 19:20:22.407206 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.407143    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b84b0ca-c62f-402a-9123-c8664d3ea056" containerName="revoke-keys"
Nov 02 19:20:22.407289 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.407228    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b84b0ca-c62f-402a-9123-c8664d3ea056" containerName="revoke-keys"
Nov 02 19:20:22.410234 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.410218    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.462246 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.462213    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:20:22.466413 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.466388    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"]
Nov 02 19:20:22.466607 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.466588    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mlq\" (UniqueName: \"kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.466686 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.466651    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.466805 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.466790    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.511921 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.511885    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:20:22.512113 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.511993    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:20:22.512113 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.512035    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:20:22.567503 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.567462    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mlq\" (UniqueName: \"kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.567709 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.567518    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.567709 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.567585    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.568257 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.568231    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.570249 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.570228    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.623886 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.623857    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mlq\" (UniqueName: \"kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq\") pod \"maas-api-revoke-sub-fa627357-hxqtq\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:22.723538 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:22.723447    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:23.101499 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:23.101475    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"]
Nov 02 19:20:23.103501 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:20:23.103463    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf02c4e_6ea7_4aa3_84cc_b00a858634bc.slice/crio-173c5b59f8777e3d7dcfea5ff12f856473d307710a14db92d9747892585c3345 WatchSource:0}: Error finding container 173c5b59f8777e3d7dcfea5ff12f856473d307710a14db92d9747892585c3345: Status 404 returned error can't find the container with id 173c5b59f8777e3d7dcfea5ff12f856473d307710a14db92d9747892585c3345
Nov 02 19:20:23.976938 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:23.976903    4476 generic.go:356] "Generic (PLEG): container finished" podID="ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" containerID="618fb2a41542ea44978270e0eafdeb989dd2dfa64abd92f09eb1e4fa9a89ffff" exitCode=0
Nov 02 19:20:23.977295 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:23.976941    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq" event={"ID":"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc","Type":"ContainerDied","Data":"618fb2a41542ea44978270e0eafdeb989dd2dfa64abd92f09eb1e4fa9a89ffff"}
Nov 02 19:20:23.977295 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:23.977003    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq" event={"ID":"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc","Type":"ContainerStarted","Data":"173c5b59f8777e3d7dcfea5ff12f856473d307710a14db92d9747892585c3345"}
Nov 02 19:20:25.117536 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.117512    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:25.189262 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.189233    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca\") pod \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") "
Nov 02 19:20:25.189430 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.189282    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq\" (UniqueName: \"kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq\") pod \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") "
Nov 02 19:20:25.189430 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.189405    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca\") pod \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\" (UID: \"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc\") "
Nov 02 19:20:25.189849 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.189819    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca" pod "ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" (UID: "ecf02c4e-6ea7-4aa3-84cc-b00a858634bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:20:25.192186 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.192159    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca" pod "ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" (UID: "ecf02c4e-6ea7-4aa3-84cc-b00a858634bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:20:25.192186 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.192183    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq" pod "ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" (UID: "ecf02c4e-6ea7-4aa3-84cc-b00a858634bc"). InnerVolumeSpecName "kube-api-access-h5mlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:20:25.290845 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.290760    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:25.290845 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.290792    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5mlq\" (UniqueName: \"kubernetes.io/projected/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-kube-api-access-h5mlq\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:25.290845 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.290802    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf02c4e-6ea7-4aa3-84cc-b00a858634bc-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:25.989993 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.989955    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq"
Nov 02 19:20:25.989993 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.989975    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa627357-hxqtq" event={"ID":"ecf02c4e-6ea7-4aa3-84cc-b00a858634bc","Type":"ContainerDied","Data":"173c5b59f8777e3d7dcfea5ff12f856473d307710a14db92d9747892585c3345"}
Nov 02 19:20:25.990258 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:25.990033    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="173c5b59f8777e3d7dcfea5ff12f856473d307710a14db92d9747892585c3345"
Nov 02 19:20:29.059558 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:20:29.059525    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:20:33.102120 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.102038    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"]
Nov 02 19:20:33.102510 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.102437    4476 cpu_manager.go:414] "containerMap: removing container" podUID="ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" containerName="revoke-keys"
Nov 02 19:20:33.102510 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.102449    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" containerName="revoke-keys"
Nov 02 19:20:33.102597 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.102544    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf02c4e-6ea7-4aa3-84cc-b00a858634bc" containerName="revoke-keys"
Nov 02 19:20:33.105956 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.105940    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.158479 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.158445    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:20:33.158685 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.158556    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:20:33.179010 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.178977    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.179177 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.179021    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.179262 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.179185    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95drx\" (UniqueName: \"kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.207933 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.207905    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:20:33.208087 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.207995    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:20:33.211295 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.211275    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"]
Nov 02 19:20:33.280541 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.280511    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95drx\" (UniqueName: \"kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.280750 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.280562    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.280750 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.280604    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.281176 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.281152    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.283167 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.283148    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.336939 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.336909    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95drx\" (UniqueName: \"kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx\") pod \"maas-api-revoke-sub-338fd32a-cxq2d\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.419184 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.419157    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:33.794456 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:33.794428    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"]
Nov 02 19:20:33.796114 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:20:33.796073    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f097bd_1b40_47c4_a272_dbed3b6bdd1b.slice/crio-1d4cc00dd108a55979cc23745ab898f5a5d1dc4111a668b57f5a1f3de86136bc WatchSource:0}: Error finding container 1d4cc00dd108a55979cc23745ab898f5a5d1dc4111a668b57f5a1f3de86136bc: Status 404 returned error can't find the container with id 1d4cc00dd108a55979cc23745ab898f5a5d1dc4111a668b57f5a1f3de86136bc
Nov 02 19:20:34.054036 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:34.054006    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d" event={"ID":"32f097bd-1b40-47c4-a272-dbed3b6bdd1b","Type":"ContainerStarted","Data":"1d4cc00dd108a55979cc23745ab898f5a5d1dc4111a668b57f5a1f3de86136bc"}
Nov 02 19:20:35.061354 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:35.061321    4476 generic.go:356] "Generic (PLEG): container finished" podID="32f097bd-1b40-47c4-a272-dbed3b6bdd1b" containerID="4af50061ae07ff92c5c8abcca19f5439c11077472cd2224011800951f4d65a26" exitCode=0
Nov 02 19:20:35.061756 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:35.061412    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d" event={"ID":"32f097bd-1b40-47c4-a272-dbed3b6bdd1b","Type":"ContainerDied","Data":"4af50061ae07ff92c5c8abcca19f5439c11077472cd2224011800951f4d65a26"}
Nov 02 19:20:36.199963 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.199936    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:36.312842 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.312810    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx\" (UniqueName: \"kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx\") pod \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") "
Nov 02 19:20:36.313042 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.312913    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca\") pod \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") "
Nov 02 19:20:36.313135 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.313086    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca\") pod \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\" (UID: \"32f097bd-1b40-47c4-a272-dbed3b6bdd1b\") "
Nov 02 19:20:36.313294 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.313269    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca" pod "32f097bd-1b40-47c4-a272-dbed3b6bdd1b" (UID: "32f097bd-1b40-47c4-a272-dbed3b6bdd1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:20:36.313796 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.313767    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:36.315893 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.315863    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx" pod "32f097bd-1b40-47c4-a272-dbed3b6bdd1b" (UID: "32f097bd-1b40-47c4-a272-dbed3b6bdd1b"). InnerVolumeSpecName "kube-api-access-95drx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:20:36.316184 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.316104    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca" pod "32f097bd-1b40-47c4-a272-dbed3b6bdd1b" (UID: "32f097bd-1b40-47c4-a272-dbed3b6bdd1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:20:36.414457 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.414415    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:36.414457 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:36.414448    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95drx\" (UniqueName: \"kubernetes.io/projected/32f097bd-1b40-47c4-a272-dbed3b6bdd1b-kube-api-access-95drx\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:37.074614 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:37.074587    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d"
Nov 02 19:20:37.074614 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:37.074598    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338fd32a-cxq2d" event={"ID":"32f097bd-1b40-47c4-a272-dbed3b6bdd1b","Type":"ContainerDied","Data":"1d4cc00dd108a55979cc23745ab898f5a5d1dc4111a668b57f5a1f3de86136bc"}
Nov 02 19:20:37.074852 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:37.074664    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4cc00dd108a55979cc23745ab898f5a5d1dc4111a668b57f5a1f3de86136bc"
Nov 02 19:20:43.853831 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.853798    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"]
Nov 02 19:20:43.854476 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.854178    4476 cpu_manager.go:414] "containerMap: removing container" podUID="32f097bd-1b40-47c4-a272-dbed3b6bdd1b" containerName="revoke-keys"
Nov 02 19:20:43.854476 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.854189    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f097bd-1b40-47c4-a272-dbed3b6bdd1b" containerName="revoke-keys"
Nov 02 19:20:43.854476 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.854274    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f097bd-1b40-47c4-a272-dbed3b6bdd1b" containerName="revoke-keys"
Nov 02 19:20:43.857410 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.857388    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:43.913987 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.913951    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"]
Nov 02 19:20:43.957943 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.957913    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:20:43.958857 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.958841    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:20:43.958857 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.958845    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:20:43.959022 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.958868    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:20:43.986256 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.986222    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:43.986455 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.986261    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:43.986455 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:43.986292    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9qf\" (UniqueName: \"kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.087735 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.087683    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.087735 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.087740    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.088059 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.087770    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9qf\" (UniqueName: \"kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.088370 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.088348    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.090318 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.090300    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.143916 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.143883    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9qf\" (UniqueName: \"kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf\") pod \"maas-api-revoke-sub-23310bb3-txkk2\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.170713 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.170683    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:44.544599 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:44.544570    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"]
Nov 02 19:20:44.546082 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:20:44.546051    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14ea395_9544_40ac_a48f_e122d16cf349.slice/crio-3457b441e0d1dd45097ed2b4b0839874f10fcfa5a6948b93ba226b236de9e0e5 WatchSource:0}: Error finding container 3457b441e0d1dd45097ed2b4b0839874f10fcfa5a6948b93ba226b236de9e0e5: Status 404 returned error can't find the container with id 3457b441e0d1dd45097ed2b4b0839874f10fcfa5a6948b93ba226b236de9e0e5
Nov 02 19:20:45.122051 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:45.122019    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2" event={"ID":"c14ea395-9544-40ac-a48f-e122d16cf349","Type":"ContainerStarted","Data":"f88940764713a7873f178062c26a94bf810251e6d5be0972ae60e1d2689b3a0a"}
Nov 02 19:20:45.122486 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:45.122076    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2" event={"ID":"c14ea395-9544-40ac-a48f-e122d16cf349","Type":"ContainerStarted","Data":"3457b441e0d1dd45097ed2b4b0839874f10fcfa5a6948b93ba226b236de9e0e5"}
Nov 02 19:20:45.233495 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:45.233434    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2" podStartSLOduration=2.233416049 podStartE2EDuration="2.233416049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:20:45.233188013 +0000 UTC m=+7076.073793946" watchObservedRunningTime="2026-11-02 19:20:45.233416049 +0000 UTC m=+7076.074021986"
Nov 02 19:20:46.129821 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:46.129786    4476 generic.go:356] "Generic (PLEG): container finished" podID="c14ea395-9544-40ac-a48f-e122d16cf349" containerID="f88940764713a7873f178062c26a94bf810251e6d5be0972ae60e1d2689b3a0a" exitCode=0
Nov 02 19:20:46.130242 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:46.129875    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2" event={"ID":"c14ea395-9544-40ac-a48f-e122d16cf349","Type":"ContainerDied","Data":"f88940764713a7873f178062c26a94bf810251e6d5be0972ae60e1d2689b3a0a"}
Nov 02 19:20:47.415081 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.407802    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:47.522982 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.522929    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf\" (UniqueName: \"kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf\") pod \"c14ea395-9544-40ac-a48f-e122d16cf349\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") "
Nov 02 19:20:47.523176 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.523046    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca\") pod \"c14ea395-9544-40ac-a48f-e122d16cf349\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") "
Nov 02 19:20:47.523176 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.523069    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca\") pod \"c14ea395-9544-40ac-a48f-e122d16cf349\" (UID: \"c14ea395-9544-40ac-a48f-e122d16cf349\") "
Nov 02 19:20:47.523666 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.523517    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca" pod "c14ea395-9544-40ac-a48f-e122d16cf349" (UID: "c14ea395-9544-40ac-a48f-e122d16cf349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:20:47.525590 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.525562    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf" pod "c14ea395-9544-40ac-a48f-e122d16cf349" (UID: "c14ea395-9544-40ac-a48f-e122d16cf349"). InnerVolumeSpecName "kube-api-access-ht9qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:20:47.527735 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.527713    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca" pod "c14ea395-9544-40ac-a48f-e122d16cf349" (UID: "c14ea395-9544-40ac-a48f-e122d16cf349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:20:47.634271 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.634171    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:47.634271 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.634215    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14ea395-9544-40ac-a48f-e122d16cf349-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:47.634271 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:47.634231    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9qf\" (UniqueName: \"kubernetes.io/projected/c14ea395-9544-40ac-a48f-e122d16cf349-kube-api-access-ht9qf\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:48.189651 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:48.186561    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2" event={"ID":"c14ea395-9544-40ac-a48f-e122d16cf349","Type":"ContainerDied","Data":"3457b441e0d1dd45097ed2b4b0839874f10fcfa5a6948b93ba226b236de9e0e5"}
Nov 02 19:20:48.189651 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:48.186650    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3457b441e0d1dd45097ed2b4b0839874f10fcfa5a6948b93ba226b236de9e0e5"
Nov 02 19:20:48.189651 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:48.186720    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23310bb3-txkk2"
Nov 02 19:20:54.605116 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.605079    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"]
Nov 02 19:20:54.605523 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.605452    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c14ea395-9544-40ac-a48f-e122d16cf349" containerName="revoke-keys"
Nov 02 19:20:54.605523 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.605463    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14ea395-9544-40ac-a48f-e122d16cf349" containerName="revoke-keys"
Nov 02 19:20:54.605601 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.605550    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14ea395-9544-40ac-a48f-e122d16cf349" containerName="revoke-keys"
Nov 02 19:20:54.608785 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.608769    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.663603 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.663571    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"]
Nov 02 19:20:54.709931 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.709903    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:20:54.710124 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.709964    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:20:54.710124 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.709976    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:20:54.710124 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.710018    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:20:54.733305 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.733275    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.733432 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.733335    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.733432 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.733411    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpgk\" (UniqueName: \"kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.834459 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.834426    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpgk\" (UniqueName: \"kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.834734 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.834497    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.834734 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.834562    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.835362 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.835341    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.837148 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.837131    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.890678 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.890648    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpgk\" (UniqueName: \"kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk\") pod \"maas-api-revoke-sub-90bc7130-kl4gs\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:54.921127 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:54.921100    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:55.296688 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:20:55.296647    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d7871e_0ee9_4117_b554_fe846535a2a5.slice/crio-ed844a1e6cea71f93898b22db5da8d5610dd2622334a64c336ebfd76a15079bb WatchSource:0}: Error finding container ed844a1e6cea71f93898b22db5da8d5610dd2622334a64c336ebfd76a15079bb: Status 404 returned error can't find the container with id ed844a1e6cea71f93898b22db5da8d5610dd2622334a64c336ebfd76a15079bb
Nov 02 19:20:55.297045 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:55.297011    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"]
Nov 02 19:20:56.237833 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:56.237795    4476 generic.go:356] "Generic (PLEG): container finished" podID="d7d7871e-0ee9-4117-b554-fe846535a2a5" containerID="7737c85fe746fdecd3740ebccd34057a6f9d5a3f16a321e78ccc77c380c1006c" exitCode=0
Nov 02 19:20:56.238258 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:56.237880    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs" event={"ID":"d7d7871e-0ee9-4117-b554-fe846535a2a5","Type":"ContainerDied","Data":"7737c85fe746fdecd3740ebccd34057a6f9d5a3f16a321e78ccc77c380c1006c"}
Nov 02 19:20:56.238258 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:56.237933    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs" event={"ID":"d7d7871e-0ee9-4117-b554-fe846535a2a5","Type":"ContainerStarted","Data":"ed844a1e6cea71f93898b22db5da8d5610dd2622334a64c336ebfd76a15079bb"}
Nov 02 19:20:57.412434 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.412404    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:57.559376 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.559281    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk\" (UniqueName: \"kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk\") pod \"d7d7871e-0ee9-4117-b554-fe846535a2a5\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") "
Nov 02 19:20:57.559580 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.559402    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca\") pod \"d7d7871e-0ee9-4117-b554-fe846535a2a5\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") "
Nov 02 19:20:57.559580 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.559543    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca\") pod \"d7d7871e-0ee9-4117-b554-fe846535a2a5\" (UID: \"d7d7871e-0ee9-4117-b554-fe846535a2a5\") "
Nov 02 19:20:57.560218 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.560183    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca" pod "d7d7871e-0ee9-4117-b554-fe846535a2a5" (UID: "d7d7871e-0ee9-4117-b554-fe846535a2a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:20:57.562412 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.562371    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk" pod "d7d7871e-0ee9-4117-b554-fe846535a2a5" (UID: "d7d7871e-0ee9-4117-b554-fe846535a2a5"). InnerVolumeSpecName "kube-api-access-xkpgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:20:57.565767 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.565734    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca" pod "d7d7871e-0ee9-4117-b554-fe846535a2a5" (UID: "d7d7871e-0ee9-4117-b554-fe846535a2a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:20:57.661136 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.661091    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:57.661136 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.661126    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7871e-0ee9-4117-b554-fe846535a2a5-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:57.661136 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:57.661142    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpgk\" (UniqueName: \"kubernetes.io/projected/d7d7871e-0ee9-4117-b554-fe846535a2a5-kube-api-access-xkpgk\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:20:58.268476 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:58.268433    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs" event={"ID":"d7d7871e-0ee9-4117-b554-fe846535a2a5","Type":"ContainerDied","Data":"ed844a1e6cea71f93898b22db5da8d5610dd2622334a64c336ebfd76a15079bb"}
Nov 02 19:20:58.268835 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:58.268815    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed844a1e6cea71f93898b22db5da8d5610dd2622334a64c336ebfd76a15079bb"
Nov 02 19:20:58.269037 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:20:58.269022    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90bc7130-kl4gs"
Nov 02 19:20:59.129972 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:20:59.129934    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:21:05.335970 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.335936    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"]
Nov 02 19:21:05.336388 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.336342    4476 cpu_manager.go:414] "containerMap: removing container" podUID="d7d7871e-0ee9-4117-b554-fe846535a2a5" containerName="revoke-keys"
Nov 02 19:21:05.336388 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.336355    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d7871e-0ee9-4117-b554-fe846535a2a5" containerName="revoke-keys"
Nov 02 19:21:05.336466 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.336426    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d7871e-0ee9-4117-b554-fe846535a2a5" containerName="revoke-keys"
Nov 02 19:21:05.339735 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.339712    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.443196 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.443148    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:21:05.443399 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.443201    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:21:05.443399 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.443210    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:21:05.443399 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.443267    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:21:05.446456 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.446439    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"]
Nov 02 19:21:05.462851 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.462827    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cftx\" (UniqueName: \"kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.462922 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.462858    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.462980 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.462955    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.563800 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.563765    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cftx\" (UniqueName: \"kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.563800 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.563798    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.564029 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.563840    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.564570 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.564550    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.566384 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.566363    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.620371 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.620335    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cftx\" (UniqueName: \"kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx\") pod \"maas-api-revoke-sub-e5518571-hlz8m\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:05.653313 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:05.653279    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:06.028084 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:06.028053    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"]
Nov 02 19:21:06.029509 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:21:06.029482    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a43fb9_5510_4d45_aa7d_bb622fedb969.slice/crio-3b8c494a690e468c26d604bdf6831641b1f7b0e5af1720ea83543bb8a5020ff6 WatchSource:0}: Error finding container 3b8c494a690e468c26d604bdf6831641b1f7b0e5af1720ea83543bb8a5020ff6: Status 404 returned error can't find the container with id 3b8c494a690e468c26d604bdf6831641b1f7b0e5af1720ea83543bb8a5020ff6
Nov 02 19:21:06.332922 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:06.332836    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m" event={"ID":"18a43fb9-5510-4d45-aa7d-bb622fedb969","Type":"ContainerStarted","Data":"3b8c494a690e468c26d604bdf6831641b1f7b0e5af1720ea83543bb8a5020ff6"}
Nov 02 19:21:07.340368 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:07.340280    4476 generic.go:356] "Generic (PLEG): container finished" podID="18a43fb9-5510-4d45-aa7d-bb622fedb969" containerID="3da185dc52f868c523f76525004fd77c33bfe2633348ed249649cdaaa698cd10" exitCode=0
Nov 02 19:21:07.340788 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:07.340367    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m" event={"ID":"18a43fb9-5510-4d45-aa7d-bb622fedb969","Type":"ContainerDied","Data":"3da185dc52f868c523f76525004fd77c33bfe2633348ed249649cdaaa698cd10"}
Nov 02 19:21:08.522094 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.522062    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:08.692037 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.691990    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca\") pod \"18a43fb9-5510-4d45-aa7d-bb622fedb969\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") "
Nov 02 19:21:08.692262 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.692057    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx\" (UniqueName: \"kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx\") pod \"18a43fb9-5510-4d45-aa7d-bb622fedb969\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") "
Nov 02 19:21:08.692262 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.692088    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca\") pod \"18a43fb9-5510-4d45-aa7d-bb622fedb969\" (UID: \"18a43fb9-5510-4d45-aa7d-bb622fedb969\") "
Nov 02 19:21:08.692734 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.692696    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca" pod "18a43fb9-5510-4d45-aa7d-bb622fedb969" (UID: "18a43fb9-5510-4d45-aa7d-bb622fedb969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:21:08.694688 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.694604    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx" pod "18a43fb9-5510-4d45-aa7d-bb622fedb969" (UID: "18a43fb9-5510-4d45-aa7d-bb622fedb969"). InnerVolumeSpecName "kube-api-access-7cftx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:21:08.695769 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.695740    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca" pod "18a43fb9-5510-4d45-aa7d-bb622fedb969" (UID: "18a43fb9-5510-4d45-aa7d-bb622fedb969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:21:08.795439 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.795401    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:08.795746 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.795722    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cftx\" (UniqueName: \"kubernetes.io/projected/18a43fb9-5510-4d45-aa7d-bb622fedb969-kube-api-access-7cftx\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:08.795882 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:08.795866    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a43fb9-5510-4d45-aa7d-bb622fedb969-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:09.425388 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:09.425352    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m"
Nov 02 19:21:09.427993 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:09.427956    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5518571-hlz8m" event={"ID":"18a43fb9-5510-4d45-aa7d-bb622fedb969","Type":"ContainerDied","Data":"3b8c494a690e468c26d604bdf6831641b1f7b0e5af1720ea83543bb8a5020ff6"}
Nov 02 19:21:09.428154 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:09.428021    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8c494a690e468c26d604bdf6831641b1f7b0e5af1720ea83543bb8a5020ff6"
Nov 02 19:21:16.121646 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.121589    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"]
Nov 02 19:21:16.122146 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.122016    4476 cpu_manager.go:414] "containerMap: removing container" podUID="18a43fb9-5510-4d45-aa7d-bb622fedb969" containerName="revoke-keys"
Nov 02 19:21:16.122146 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.122030    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a43fb9-5510-4d45-aa7d-bb622fedb969" containerName="revoke-keys"
Nov 02 19:21:16.122146 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.122090    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a43fb9-5510-4d45-aa7d-bb622fedb969" containerName="revoke-keys"
Nov 02 19:21:16.125500 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.125483    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.177893 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.177863    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.178118 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.177933    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.178118 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.178086    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdps6\" (UniqueName: \"kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.181316 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.181285    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"]
Nov 02 19:21:16.226370 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.226338    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:21:16.226551 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.226393    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:21:16.226888 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.226871    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:21:16.226937 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.226876    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:21:16.279272 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.279245    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.279467 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.279295    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdps6\" (UniqueName: \"kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.279467 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.279338    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.280040 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.280012    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.281926 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.281903    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.335344 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.335313    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdps6\" (UniqueName: \"kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6\") pod \"maas-api-revoke-sub-49a4969b-lwgdt\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.439786 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.439705    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:16.818370 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:16.818341    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"]
Nov 02 19:21:16.819449 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:21:16.819417    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37dd32a_651e_44bc_ae79_f4db93408878.slice/crio-ca06623e686e10baa18c53f8c4067de7b13808ca954072718c2a229a168f2c13 WatchSource:0}: Error finding container ca06623e686e10baa18c53f8c4067de7b13808ca954072718c2a229a168f2c13: Status 404 returned error can't find the container with id ca06623e686e10baa18c53f8c4067de7b13808ca954072718c2a229a168f2c13
Nov 02 19:21:17.471493 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:17.471408    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt" event={"ID":"c37dd32a-651e-44bc-ae79-f4db93408878","Type":"ContainerStarted","Data":"7ee8671f040ceb779323cb2b6aa35885e471bad27fa923a0765c2db69d831c18"}
Nov 02 19:21:17.471493 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:17.471462    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt" event={"ID":"c37dd32a-651e-44bc-ae79-f4db93408878","Type":"ContainerStarted","Data":"ca06623e686e10baa18c53f8c4067de7b13808ca954072718c2a229a168f2c13"}
Nov 02 19:21:17.582097 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:17.582030    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt" podStartSLOduration=1.582013757 podStartE2EDuration="1.582013757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:21:17.581790272 +0000 UTC m=+7108.422396207" watchObservedRunningTime="2026-11-02 19:21:17.582013757 +0000 UTC m=+7108.422619691"
Nov 02 19:21:18.479317 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:18.479282    4476 generic.go:356] "Generic (PLEG): container finished" podID="c37dd32a-651e-44bc-ae79-f4db93408878" containerID="7ee8671f040ceb779323cb2b6aa35885e471bad27fa923a0765c2db69d831c18" exitCode=0
Nov 02 19:21:18.479795 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:18.479373    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt" event={"ID":"c37dd32a-651e-44bc-ae79-f4db93408878","Type":"ContainerDied","Data":"7ee8671f040ceb779323cb2b6aa35885e471bad27fa923a0765c2db69d831c18"}
Nov 02 19:21:20.012836 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.012387    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:20.029319 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.029287    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6\" (UniqueName: \"kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6\") pod \"c37dd32a-651e-44bc-ae79-f4db93408878\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") "
Nov 02 19:21:20.029540 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.029356    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca\") pod \"c37dd32a-651e-44bc-ae79-f4db93408878\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") "
Nov 02 19:21:20.029540 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.029401    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca\") pod \"c37dd32a-651e-44bc-ae79-f4db93408878\" (UID: \"c37dd32a-651e-44bc-ae79-f4db93408878\") "
Nov 02 19:21:20.030444 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.030417    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca" pod "c37dd32a-651e-44bc-ae79-f4db93408878" (UID: "c37dd32a-651e-44bc-ae79-f4db93408878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:21:20.033058 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.033030    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6" pod "c37dd32a-651e-44bc-ae79-f4db93408878" (UID: "c37dd32a-651e-44bc-ae79-f4db93408878"). InnerVolumeSpecName "kube-api-access-hdps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:21:20.033388 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.033367    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca" pod "c37dd32a-651e-44bc-ae79-f4db93408878" (UID: "c37dd32a-651e-44bc-ae79-f4db93408878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:21:20.132250 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.132216    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:20.132250 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.132251    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37dd32a-651e-44bc-ae79-f4db93408878-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:20.132476 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.132265    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdps6\" (UniqueName: \"kubernetes.io/projected/c37dd32a-651e-44bc-ae79-f4db93408878-kube-api-access-hdps6\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:20.525811 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.525782    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt"
Nov 02 19:21:20.525995 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.525829    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a4969b-lwgdt" event={"ID":"c37dd32a-651e-44bc-ae79-f4db93408878","Type":"ContainerDied","Data":"ca06623e686e10baa18c53f8c4067de7b13808ca954072718c2a229a168f2c13"}
Nov 02 19:21:20.525995 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:20.525893    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca06623e686e10baa18c53f8c4067de7b13808ca954072718c2a229a168f2c13"
Nov 02 19:21:26.045609 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.045571    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"]
Nov 02 19:21:26.094959 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.094932    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08a949cf-x2hf2"]
Nov 02 19:21:26.878168 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.878137    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"]
Nov 02 19:21:26.878531 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.878518    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c37dd32a-651e-44bc-ae79-f4db93408878" containerName="revoke-keys"
Nov 02 19:21:26.878581 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.878532    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37dd32a-651e-44bc-ae79-f4db93408878" containerName="revoke-keys"
Nov 02 19:21:26.878654 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.878606    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37dd32a-651e-44bc-ae79-f4db93408878" containerName="revoke-keys"
Nov 02 19:21:26.881701 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.881685    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:26.919922 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.919899    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:26.920067 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.919951    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s72c\" (UniqueName: \"kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:26.920067 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.920021    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:26.934101 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.934074    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:21:26.938646 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.938603    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"]
Nov 02 19:21:26.983352 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.983322    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:21:26.983352 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.983331    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:21:26.983571 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:26.983408    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:21:27.020536 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.020509    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.020726 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.020574    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s72c\" (UniqueName: \"kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.020726 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.020602    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.021209 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.021192    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.023166 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.023145    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.076680 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.076643    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s72c\" (UniqueName: \"kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c\") pod \"maas-api-revoke-sub-94ece844-hrv94\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.194849 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.194759    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:27.382091 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.382056    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e87d3f-88ea-4a10-8e10-553cc86786a2" path="/var/lib/kubelet/pods/f9e87d3f-88ea-4a10-8e10-553cc86786a2/volumes"
Nov 02 19:21:27.574106 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:27.574081    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"]
Nov 02 19:21:27.575304 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:21:27.575276    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a7e9a3_5567_4a2c_a9d6_cdfa764a25b9.slice/crio-c2d662cf44e13f860cc12b60cdd4233319f8507dfdb1a1c1f4a30076e5be3440 WatchSource:0}: Error finding container c2d662cf44e13f860cc12b60cdd4233319f8507dfdb1a1c1f4a30076e5be3440: Status 404 returned error can't find the container with id c2d662cf44e13f860cc12b60cdd4233319f8507dfdb1a1c1f4a30076e5be3440
Nov 02 19:21:28.573209 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:28.573174    4476 generic.go:356] "Generic (PLEG): container finished" podID="c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" containerID="384e92fcbdc3ac653e4763036c250fad1cf17f074f1969199a995a797b13e9ba" exitCode=0
Nov 02 19:21:28.573645 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:28.573272    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94" event={"ID":"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9","Type":"ContainerDied","Data":"384e92fcbdc3ac653e4763036c250fad1cf17f074f1969199a995a797b13e9ba"}
Nov 02 19:21:28.573645 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:28.573326    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94" event={"ID":"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9","Type":"ContainerStarted","Data":"c2d662cf44e13f860cc12b60cdd4233319f8507dfdb1a1c1f4a30076e5be3440"}
Nov 02 19:21:29.059648 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:21:29.059594    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:21:29.748157 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.748124    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:29.845816 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.845715    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca\") pod \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") "
Nov 02 19:21:29.846008 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.845828    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c\" (UniqueName: \"kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c\") pod \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") "
Nov 02 19:21:29.846008 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.845857    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca\") pod \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\" (UID: \"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9\") "
Nov 02 19:21:29.846218 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.846190    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca" pod "c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" (UID: "c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:21:29.848256 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.848228    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca" pod "c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" (UID: "c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:21:29.848741 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.848706    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c" pod "c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" (UID: "c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9"). InnerVolumeSpecName "kube-api-access-8s72c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:21:29.947152 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.947099    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s72c\" (UniqueName: \"kubernetes.io/projected/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-kube-api-access-8s72c\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:29.947152 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.947142    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:29.947152 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:29.947158    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:30.602653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:30.601674    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94" event={"ID":"c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9","Type":"ContainerDied","Data":"c2d662cf44e13f860cc12b60cdd4233319f8507dfdb1a1c1f4a30076e5be3440"}
Nov 02 19:21:30.602653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:30.601759    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d662cf44e13f860cc12b60cdd4233319f8507dfdb1a1c1f4a30076e5be3440"
Nov 02 19:21:30.602653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:30.601838    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94ece844-hrv94"
Nov 02 19:21:37.053026 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.052985    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"]
Nov 02 19:21:37.100290 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.100250    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d65a4e6e-z7gt5"]
Nov 02 19:21:37.381827 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.381795    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b81b6c-104c-42a1-b977-b62c696d6b38" path="/var/lib/kubelet/pods/80b81b6c-104c-42a1-b977-b62c696d6b38/volumes"
Nov 02 19:21:37.674934 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.674848    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"]
Nov 02 19:21:37.675347 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.675329    4476 cpu_manager.go:414] "containerMap: removing container" podUID="c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" containerName="revoke-keys"
Nov 02 19:21:37.675347 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.675349    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" containerName="revoke-keys"
Nov 02 19:21:37.675521 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.675471    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9" containerName="revoke-keys"
Nov 02 19:21:37.679796 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.679773    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.734726 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.734699    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"]
Nov 02 19:21:37.779382 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.779352    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:21:37.779542 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.779448    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:21:37.779542 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.779511    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:21:37.781061 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.781044    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:21:37.843453 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.843408    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.843698 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.843477    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rn8h\" (UniqueName: \"kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.843698 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.843602    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.944868 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.944771    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.944868 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.944837    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rn8h\" (UniqueName: \"kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.945135 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.944901    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.945587 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.945566    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:37.947336 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:37.947314    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:38.001674 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:38.001610    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rn8h\" (UniqueName: \"kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h\") pod \"maas-api-revoke-sub-cef039c0-4ptfw\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:38.293126 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:38.293026    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:38.670245 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:38.670220    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"]
Nov 02 19:21:38.672303 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:21:38.672278    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d9dfa1_7f3d_42f8_9d50_7ea2de37a3a7.slice/crio-6ec457632c4fe93a754dbd7498b5ea90b36aa2cbddd20ef6eb68136d7db586fc WatchSource:0}: Error finding container 6ec457632c4fe93a754dbd7498b5ea90b36aa2cbddd20ef6eb68136d7db586fc: Status 404 returned error can't find the container with id 6ec457632c4fe93a754dbd7498b5ea90b36aa2cbddd20ef6eb68136d7db586fc
Nov 02 19:21:39.662859 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:39.662828    4476 generic.go:356] "Generic (PLEG): container finished" podID="e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" containerID="bf203238fdf11ec42b1b5f20386065a2469911fa795653bfbe2fc04b8d4d7321" exitCode=0
Nov 02 19:21:39.663260 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:39.662919    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw" event={"ID":"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7","Type":"ContainerDied","Data":"bf203238fdf11ec42b1b5f20386065a2469911fa795653bfbe2fc04b8d4d7321"}
Nov 02 19:21:39.663260 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:39.662971    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw" event={"ID":"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7","Type":"ContainerStarted","Data":"6ec457632c4fe93a754dbd7498b5ea90b36aa2cbddd20ef6eb68136d7db586fc"}
Nov 02 19:21:40.847903 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.847874    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:40.969601 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.969505    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h\" (UniqueName: \"kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h\") pod \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") "
Nov 02 19:21:40.969817 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.969617    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca\") pod \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") "
Nov 02 19:21:40.969817 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.969704    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca\") pod \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\" (UID: \"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7\") "
Nov 02 19:21:40.970434 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.970333    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca" pod "e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" (UID: "e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:21:40.972854 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.972826    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca" pod "e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" (UID: "e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:21:40.973029 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:40.972926    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h" pod "e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" (UID: "e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7"). InnerVolumeSpecName "kube-api-access-5rn8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:21:41.070645 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:41.070589    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rn8h\" (UniqueName: \"kubernetes.io/projected/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-kube-api-access-5rn8h\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:41.070840 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:41.070672    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:41.070840 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:41.070688    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:41.716733 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:41.716346    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw" event={"ID":"e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7","Type":"ContainerDied","Data":"6ec457632c4fe93a754dbd7498b5ea90b36aa2cbddd20ef6eb68136d7db586fc"}
Nov 02 19:21:41.716733 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:41.716411    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ec457632c4fe93a754dbd7498b5ea90b36aa2cbddd20ef6eb68136d7db586fc"
Nov 02 19:21:41.716733 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:41.716479    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cef039c0-4ptfw"
Nov 02 19:21:48.465331 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.465296    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"]
Nov 02 19:21:48.466037 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.465706    4476 cpu_manager.go:414] "containerMap: removing container" podUID="e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" containerName="revoke-keys"
Nov 02 19:21:48.466037 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.465718    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" containerName="revoke-keys"
Nov 02 19:21:48.466037 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.465782    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7" containerName="revoke-keys"
Nov 02 19:21:48.469120 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.469098    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.472496 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.472461    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.472647 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.472539    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nr4z\" (UniqueName: \"kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.472770 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.472745    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.521736 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.521698    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 19:21:48.524112 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.524070    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"]
Nov 02 19:21:48.569740 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.569702    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\""
Nov 02 19:21:48.569740 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.569723    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 19:21:48.569740 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.569716    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 19:21:48.574140 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.574114    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.574271 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.574155    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nr4z\" (UniqueName: \"kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.574271 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.574210    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.574865 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.574834    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.576745 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.576718    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.633315 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.633283    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nr4z\" (UniqueName: \"kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z\") pod \"maas-api-revoke-sub-b36ab56e-n2lpg\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:48.783810 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:48.783728    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:49.161706 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:49.161665    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"]
Nov 02 19:21:49.167855 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:21:49.167820    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35acbdaf_af28_4782_b3b7_fcbe6fb1b466.slice/crio-66e551cfed290b580308dfe972c7063472ccd9c659c218fe5f73353da2a525f4 WatchSource:0}: Error finding container 66e551cfed290b580308dfe972c7063472ccd9c659c218fe5f73353da2a525f4: Status 404 returned error can't find the container with id 66e551cfed290b580308dfe972c7063472ccd9c659c218fe5f73353da2a525f4
Nov 02 19:21:49.769650 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:49.769533    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg" event={"ID":"35acbdaf-af28-4782-b3b7-fcbe6fb1b466","Type":"ContainerStarted","Data":"1aec4ba7df8e4402669e22797f373aad03570a139e5a795703c3bbb5f5993ab1"}
Nov 02 19:21:49.769650 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:49.769590    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg" event={"ID":"35acbdaf-af28-4782-b3b7-fcbe6fb1b466","Type":"ContainerStarted","Data":"66e551cfed290b580308dfe972c7063472ccd9c659c218fe5f73353da2a525f4"}
Nov 02 19:21:49.883310 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:49.883245    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg" podStartSLOduration=1.883224534 podStartE2EDuration="1.883224534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:21:49.881573751 +0000 UTC m=+7140.722179686" watchObservedRunningTime="2026-11-02 19:21:49.883224534 +0000 UTC m=+7140.723830470"
Nov 02 19:21:50.776767 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:50.776731    4476 generic.go:356] "Generic (PLEG): container finished" podID="35acbdaf-af28-4782-b3b7-fcbe6fb1b466" containerID="1aec4ba7df8e4402669e22797f373aad03570a139e5a795703c3bbb5f5993ab1" exitCode=0
Nov 02 19:21:50.777178 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:50.776814    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg" event={"ID":"35acbdaf-af28-4782-b3b7-fcbe6fb1b466","Type":"ContainerDied","Data":"1aec4ba7df8e4402669e22797f373aad03570a139e5a795703c3bbb5f5993ab1"}
Nov 02 19:21:52.201262 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.198820    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:52.299653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.298822    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:21:52.299653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.298894    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hlj6g" podUID="c8df61b1-f46f-4ee7-be82-3b503d5e2969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:21:52.337661 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.332764    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z\" (UniqueName: \"kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z\") pod \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") "
Nov 02 19:21:52.337661 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.332846    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca\") pod \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") "
Nov 02 19:21:52.337661 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.332902    4476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca\") pod \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\" (UID: \"35acbdaf-af28-4782-b3b7-fcbe6fb1b466\") "
Nov 02 19:21:52.344426 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.344369    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca" pod "35acbdaf-af28-4782-b3b7-fcbe6fb1b466" (UID: "35acbdaf-af28-4782-b3b7-fcbe6fb1b466"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 19:21:52.379601 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.379023    4476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 19:21:52.379601 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.379081    4476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8v2v2" podUID="d62c51c1-59c7-4553-bf45-ec662a32b98e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 19:21:52.387734 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.380535    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z" pod "35acbdaf-af28-4782-b3b7-fcbe6fb1b466" (UID: "35acbdaf-af28-4782-b3b7-fcbe6fb1b466"). InnerVolumeSpecName "kube-api-access-5nr4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 19:21:52.435653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.435180    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-openshift-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:52.435653 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.435229    4476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nr4z\" (UniqueName: \"kubernetes.io/projected/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-kube-api-access-5nr4z\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:52.511806 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.511763    4476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca" pod "35acbdaf-af28-4782-b3b7-fcbe6fb1b466" (UID: "35acbdaf-af28-4782-b3b7-fcbe6fb1b466"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 19:21:52.536356 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.536323    4476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35acbdaf-af28-4782-b3b7-fcbe6fb1b466-maas-api-xks-ca\") on node \"ip-10-0-136-192.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 19:21:52.816149 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.816061    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg" event={"ID":"35acbdaf-af28-4782-b3b7-fcbe6fb1b466","Type":"ContainerDied","Data":"66e551cfed290b580308dfe972c7063472ccd9c659c218fe5f73353da2a525f4"}
Nov 02 19:21:52.816149 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.816127    4476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e551cfed290b580308dfe972c7063472ccd9c659c218fe5f73353da2a525f4"
Nov 02 19:21:52.816448 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:21:52.816199    4476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b36ab56e-n2lpg"
Nov 02 19:21:59.058224 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:21:59.058181    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:22:10.125817 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:10.125785    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"]
Nov 02 19:22:10.129072 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:10.129043    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51954f6a-7bvmw"]
Nov 02 19:22:11.383502 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:11.383456    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90eac55-6e54-4ff2-a1a3-58a5947e767f" path="/var/lib/kubelet/pods/e90eac55-6e54-4ff2-a1a3-58a5947e767f/volumes"
Nov 02 19:22:16.108265 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:16.108215    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-685c598f54-4cmqn_e027aa27-0b9b-4c21-8e23-54644e537f96/authorino/0.log"
Nov 02 19:22:28.736857 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:28.736825    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ws6kb_13454cdd-8e2d-4c3c-92cf-0c6adc13355f/manager/0.log"
Nov 02 19:22:29.058763 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:22:29.058678    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:22:29.290616 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:29.290577    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zt8p9_dee2ee5b-9244-4eaf-8ed6-e72a261d01f7/manager/0.log"
Nov 02 19:22:30.253709 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:30.253673    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w47j_76341d23-9cf8-4a8b-becb-16a32e0c9775/server/0.log"
Nov 02 19:22:30.490501 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:30.490470    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f9fdr_56c7fe1e-8ea8-454c-9ce4-2585f83c60b6/manager/0.log"
Nov 02 19:22:32.347837 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:32.347809    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-20ab545f-r5gvw_52694be7-697d-44f3-9270-c54675228da4/revoke-keys/0.log"
Nov 02 19:22:32.557285 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:32.557255    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-23310bb3-txkk2_c14ea395-9544-40ac-a48f-e122d16cf349/revoke-keys/0.log"
Nov 02 19:22:32.772013 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:32.771984    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2daade54-vvmqb_ae553987-3811-4165-8e37-8be1e15f82da/revoke-keys/0.log"
Nov 02 19:22:32.995531 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:32.995507    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-338fd32a-cxq2d_32f097bd-1b40-47c4-a272-dbed3b6bdd1b/revoke-keys/0.log"
Nov 02 19:22:33.220354 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:33.220322    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-49a4969b-lwgdt_c37dd32a-651e-44bc-ae79-f4db93408878/revoke-keys/0.log"
Nov 02 19:22:33.432699 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:33.432668    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-659ba7af-h7fz4_6f1b0b76-e4a1-42aa-ae91-12fdb1651188/revoke-keys/0.log"
Nov 02 19:22:33.656081 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:33.656054    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-90bc7130-kl4gs_d7d7871e-0ee9-4117-b554-fe846535a2a5/revoke-keys/0.log"
Nov 02 19:22:33.869235 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:33.869201    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94ece844-hrv94_c9a7e9a3-5567-4a2c-a9d6-cdfa764a25b9/revoke-keys/0.log"
Nov 02 19:22:34.085042 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:34.084958    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ac5a7d6c-rlzmt_885728d4-e83d-4da3-b8e7-1bd1ab32bded/revoke-keys/0.log"
Nov 02 19:22:34.297429 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:34.297394    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b36ab56e-n2lpg_35acbdaf-af28-4782-b3b7-fcbe6fb1b466/revoke-keys/0.log"
Nov 02 19:22:34.521146 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:34.521112    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cef039c0-4ptfw_e7d9dfa1-7f3d-42f8-9d50-7ea2de37a3a7/revoke-keys/0.log"
Nov 02 19:22:34.742983 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:34.742945    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d56aca28-nsnz2_72781957-d260-4abe-a426-17d5461cf259/revoke-keys/0.log"
Nov 02 19:22:34.963529 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:34.963500    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e5518571-hlz8m_18a43fb9-5510-4d45-aa7d-bb622fedb969/revoke-keys/0.log"
Nov 02 19:22:35.181144 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:35.181116    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee0cb899-mclc2_4e5e36b0-5984-43c4-bbec-c045bea98184/revoke-keys/0.log"
Nov 02 19:22:35.388180 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:35.388152    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fa627357-hxqtq_ecf02c4e-6ea7-4aa3-84cc-b00a858634bc/revoke-keys/0.log"
Nov 02 19:22:35.596294 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:35.596268    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fb0b5d82-2ffpl_0b84b0ca-c62f-402a-9123-c8664d3ea056/revoke-keys/0.log"
Nov 02 19:22:37.743158 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:37.743118    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-685c598f54-4cmqn_e027aa27-0b9b-4c21-8e23-54644e537f96/authorino/0.log"
Nov 02 19:22:38.145942 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:38.145916    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rjmsd_4c792d4f-f412-4cc5-8e8f-07c797c31015/manager/0.log"
Nov 02 19:22:38.379274 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:38.379245    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hw69v_c0d81e3f-d92e-4a06-adab-a62d740747a0/manager/0.log"
Nov 02 19:22:38.593571 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:38.593492    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wlcdx_f4f28bac-e0b7-41dc-b254-0a3bdd5f997c/kuadrant-console-plugin/0.log"
Nov 02 19:22:38.973767 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:38.973724    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2rbj7_8b1fabde-e5ef-45af-9075-caef5e17ada1/manager/0.log"
Nov 02 19:22:39.466341 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:39.466306    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wg9xn_93dd06a4-ba2c-474a-8a39-c3e591131472/manager/0.log"
Nov 02 19:22:40.464404 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:40.464362    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f78nkx_42b3b731-576e-426f-9f35-7ddf92d288ab/istio-proxy/0.log"
Nov 02 19:22:42.382590 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:42.382554    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8v2v2_d62c51c1-59c7-4553-bf45-ec662a32b98e/istio-proxy/0.log"
Nov 02 19:22:42.873073 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:42.873038    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-hlj6g_c8df61b1-f46f-4ee7-be82-3b503d5e2969/istio-proxy/0.log"
Nov 02 19:22:43.654982 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:43.654951    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-thm4f_792eac8f-2598-4b49-94d1-6d0231efb074/istio-proxy/0.log"
Nov 02 19:22:43.926336 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:43.926261    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc_54b985a5-9c49-4824-9642-5f9f2fa5d66e/istio-proxy/0.log"
Nov 02 19:22:50.562645 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:50.562608    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:22:50.590906 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:50.590881    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bd79d_c304518f-84c5-4ec3-ac17-f79233dd87d4/extract/0.log"
Nov 02 19:22:52.348703 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:52.348669    4476 scope.go:122] "RemoveContainer" containerID="4d3c3b32f4be8ac872cba4fa74344bf0993cd4089d0e36d0ea96fdfc78069d46"
Nov 02 19:22:52.361440 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:52.361414    4476 scope.go:122] "RemoveContainer" containerID="dc7d4fc8edf9bb6acf21952e2f297b4736698fdab1f45d9fe2381db876d1e288"
Nov 02 19:22:55.045103 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:55.045068    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"]
Nov 02 19:22:55.094092 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:55.094059    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659ba7af-h7fz4"]
Nov 02 19:22:55.381463 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:22:55.381419    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1b0b76-e4a1-42aa-ae91-12fdb1651188" path="/var/lib/kubelet/pods/6f1b0b76-e4a1-42aa-ae91-12fdb1651188/volumes"
Nov 02 19:22:59.058195 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:22:59.058161    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:23:06.050234 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:06.050199    4476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"]
Nov 02 19:23:06.099197 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:06.099164    4476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0cb899-mclc2"]
Nov 02 19:23:07.381948 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:07.381898    4476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5e36b0-5984-43c4-bbec-c045bea98184" path="/var/lib/kubelet/pods/4e5e36b0-5984-43c4-bbec-c045bea98184/volumes"
Nov 02 19:23:08.521108 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:08.521076    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-trcmz_6f939783-c4a2-4931-9ab9-07f56e14a7ed/global-pull-secret-syncer/0.log"
Nov 02 19:23:10.475573 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:10.475539    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dz5j7_3f80bf07-9f23-46af-b53f-35bbd3157b6d/konnectivity-agent/0.log"
Nov 02 19:23:11.320689 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:11.320658    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ws6kb_13454cdd-8e2d-4c3c-92cf-0c6adc13355f/manager/0.log"
Nov 02 19:23:12.532636 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:12.532593    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zt8p9_dee2ee5b-9244-4eaf-8ed6-e72a261d01f7/manager/0.log"
Nov 02 19:23:12.763712 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:12.763686    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-192.us-west-1.compute.internal_2661666f349d448cd741b0565fc226b6/haproxy/0.log"
Nov 02 19:23:12.861000 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:12.860977    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w47j_76341d23-9cf8-4a8b-becb-16a32e0c9775/server/0.log"
Nov 02 19:23:15.028802 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:15.028770    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f9fdr_56c7fe1e-8ea8-454c-9ce4-2585f83c60b6/manager/0.log"
Nov 02 19:23:17.448834 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:17.448801    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-685c598f54-4cmqn_e027aa27-0b9b-4c21-8e23-54644e537f96/authorino/0.log"
Nov 02 19:23:18.596932 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.596895    4476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"]
Nov 02 19:23:18.597356 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.597289    4476 cpu_manager.go:414] "containerMap: removing container" podUID="35acbdaf-af28-4782-b3b7-fcbe6fb1b466" containerName="revoke-keys"
Nov 02 19:23:18.597356 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.597300    4476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35acbdaf-af28-4782-b3b7-fcbe6fb1b466" containerName="revoke-keys"
Nov 02 19:23:18.597434 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.597369    4476 memory_manager.go:365] "RemoveStaleState removing state" podUID="35acbdaf-af28-4782-b3b7-fcbe6fb1b466" containerName="revoke-keys"
Nov 02 19:23:18.600380 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.600363    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.652397 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.652360    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdnvs\"/\"kube-root-ca.crt\""
Nov 02 19:23:18.684230 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.684199    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rjmsd_4c792d4f-f412-4cc5-8e8f-07c797c31015/manager/0.log"
Nov 02 19:23:18.701604 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.701577    4476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdnvs\"/\"openshift-service-ca.crt\""
Nov 02 19:23:18.701604 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.701592    4476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pdnvs\"/\"default-dockercfg-hdbkh\""
Nov 02 19:23:18.705083 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.705058    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"]
Nov 02 19:23:18.708781 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.708758    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-proc\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.708889 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.708809    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-lib-modules\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.708953 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.708893    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-podres\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.709014 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.708965    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4hs\" (UniqueName: \"kubernetes.io/projected/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-kube-api-access-6k4hs\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.709079 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.709063    4476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-sys\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.809729 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809689    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4hs\" (UniqueName: \"kubernetes.io/projected/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-kube-api-access-6k4hs\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.809926 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809758    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-sys\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.809926 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809814    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-proc\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.809926 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809853    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-lib-modules\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.809926 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809871    4476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-podres\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.809926 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809888    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-sys\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.810140 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809959    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-podres\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.810140 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809974    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-lib-modules\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.810140 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.809959    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-proc\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.836312 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.836284    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ws6kb_13454cdd-8e2d-4c3c-92cf-0c6adc13355f/manager/0.log"
Nov 02 19:23:18.865613 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.865579    4476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4hs\" (UniqueName: \"kubernetes.io/projected/b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3-kube-api-access-6k4hs\") pod \"perf-node-gather-daemonset-kk68x\" (UID: \"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:18.916351 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:18.916323    4476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:19.100519 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.100479    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hw69v_c0d81e3f-d92e-4a06-adab-a62d740747a0/manager/0.log"
Nov 02 19:23:19.291020 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.290978    4476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"]
Nov 02 19:23:19.292464 ip-10-0-136-192 kubenswrapper[4476]: W1102 19:23:19.292431    4476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1c9c56f_1e83_49b4_8e30_1db1a1d51fb3.slice/crio-6780f282de20474871f7933433b8a8cabafb3bb9890e2bc29dce2d92b677b047 WatchSource:0}: Error finding container 6780f282de20474871f7933433b8a8cabafb3bb9890e2bc29dce2d92b677b047: Status 404 returned error can't find the container with id 6780f282de20474871f7933433b8a8cabafb3bb9890e2bc29dce2d92b677b047
Nov 02 19:23:19.412950 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.412906    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x" event={"ID":"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3","Type":"ContainerStarted","Data":"9c4ff493e805d250729a703e0e0618282b0b1640ebb6a838acc791b5236ccfa6"}
Nov 02 19:23:19.413141 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.412973    4476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x" event={"ID":"b1c9c56f-1e83-49b4-8e30-1db1a1d51fb3","Type":"ContainerStarted","Data":"6780f282de20474871f7933433b8a8cabafb3bb9890e2bc29dce2d92b677b047"}
Nov 02 19:23:19.413141 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.413003    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:19.527805 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.527750    4476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x" podStartSLOduration=1.527731975 podStartE2EDuration="1.527731975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:23:19.523914593 +0000 UTC m=+7230.364520532" watchObservedRunningTime="2026-11-02 19:23:19.527731975 +0000 UTC m=+7230.368337910"
Nov 02 19:23:19.528686 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:19.528661    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wlcdx_f4f28bac-e0b7-41dc-b254-0a3bdd5f997c/kuadrant-console-plugin/0.log"
Nov 02 19:23:20.038095 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:20.038054    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2rbj7_8b1fabde-e5ef-45af-9075-caef5e17ada1/manager/0.log"
Nov 02 19:23:20.055127 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:20.055096    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zt8p9_dee2ee5b-9244-4eaf-8ed6-e72a261d01f7/manager/0.log"
Nov 02 19:23:20.416985 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:20.416958    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w47j_76341d23-9cf8-4a8b-becb-16a32e0c9775/server/0.log"
Nov 02 19:23:21.002565 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:21.002523    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-wg9xn_93dd06a4-ba2c-474a-8a39-c3e591131472/manager/0.log"
Nov 02 19:23:22.610465 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:22.610433    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f9fdr_56c7fe1e-8ea8-454c-9ce4-2585f83c60b6/manager/0.log"
Nov 02 19:23:25.429573 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:25.429544    4476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-kk68x"
Nov 02 19:23:25.971778 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:25.971739    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ws6kb_13454cdd-8e2d-4c3c-92cf-0c6adc13355f/manager/0.log"
Nov 02 19:23:27.171915 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:27.171879    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zt8p9_dee2ee5b-9244-4eaf-8ed6-e72a261d01f7/manager/0.log"
Nov 02 19:23:27.516896 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:27.516818    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w47j_76341d23-9cf8-4a8b-becb-16a32e0c9775/server/0.log"
Nov 02 19:23:29.047301 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:29.047271    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-45wxf_a778c105-00fa-4d5e-a614-2a12c6c5b9e7/node-exporter/0.log"
Nov 02 19:23:29.066508 ip-10-0-136-192 kubenswrapper[4476]: E1102 19:23:29.066470    4476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pmnsd" containerName="extractor"
Nov 02 19:23:29.463523 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:29.463487    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-45wxf_a778c105-00fa-4d5e-a614-2a12c6c5b9e7/kube-rbac-proxy/0.log"
Nov 02 19:23:29.720081 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:29.719999    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-f9fdr_56c7fe1e-8ea8-454c-9ce4-2585f83c60b6/manager/0.log"
Nov 02 19:23:29.878973 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:29.878945    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-45wxf_a778c105-00fa-4d5e-a614-2a12c6c5b9e7/init-textfile/0.log"
Nov 02 19:23:33.327313 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:33.327226    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f78nkx_42b3b731-576e-426f-9f35-7ddf92d288ab/istio-proxy/0.log"
Nov 02 19:23:36.948518 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:36.948480    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-8v2v2_d62c51c1-59c7-4553-bf45-ec662a32b98e/istio-proxy/0.log"
Nov 02 19:23:37.382202 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:37.382171    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-hlj6g_c8df61b1-f46f-4ee7-be82-3b503d5e2969/istio-proxy/0.log"
Nov 02 19:23:38.024233 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:38.024205    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-thm4f_792eac8f-2598-4b49-94d1-6d0231efb074/istio-proxy/0.log"
Nov 02 19:23:38.232873 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:38.232832    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6c77f4c8c9-tqclc_54b985a5-9c49-4824-9642-5f9f2fa5d66e/istio-proxy/0.log"
Nov 02 19:23:39.506220 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:39.506192    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd567549c-cbcbb_17bfcc67-9352-4d82-897a-86a6c72cbf0c/thanos-query/0.log"
Nov 02 19:23:39.921101 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:39.921064    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd567549c-cbcbb_17bfcc67-9352-4d82-897a-86a6c72cbf0c/kube-rbac-proxy-web/0.log"
Nov 02 19:23:40.337521 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:40.337443    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd567549c-cbcbb_17bfcc67-9352-4d82-897a-86a6c72cbf0c/kube-rbac-proxy/0.log"
Nov 02 19:23:40.751870 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:40.751838    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd567549c-cbcbb_17bfcc67-9352-4d82-897a-86a6c72cbf0c/prom-label-proxy/0.log"
Nov 02 19:23:41.167742 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:41.167711    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd567549c-cbcbb_17bfcc67-9352-4d82-897a-86a6c72cbf0c/kube-rbac-proxy-rules/0.log"
Nov 02 19:23:41.583533 ip-10-0-136-192 kubenswrapper[4476]: I1102 19:23:41.583441    4476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bd567549c-cbcbb_17bfcc67-9352-4d82-897a-86a6c72cbf0c/kube-rbac-proxy-metrics/0.log"